Event-Triggered Single-Network ADP for Zero-Sum Game of Unknown Nonlinear Systems with Constrained Input

Abstract

In this paper, an event-triggered adaptive dynamic programming (ADP) method is proposed to deal with the $H_{\infty }$ problem with unknown dynamic and constrained input. Firstly, the $H_{\infty }$-constrained problem is regarded as the two-player zero-sum game with the nonquadratic value function. Secondly, we develop the event-triggered Hamilton–Jacobi–Isaacs(HJI) equation, and an event-triggered ADP method is proposed to solve the HJI equation, which is equivalent to solving the Nash saddle point of the zero-sum game. An event-based single-critic neural network (NN) is applied to obtain the optimal value function, which reduces the communication resource and computational cost of algorithm implementation. For the event-triggered control, a triggering condition with the level of disturbance attenuation is developed to limit the number of sampling states, and the condition avoids Zeno behavior by proving the existence of events with minimum triggering interval. It is proved theoretically that the closed-loop system is asymptotically stable, and the critic NN weight error is uniformly ultimately boundedness (UUB). The learning performance of the proposed algorithm is verified by two examples.

1. Introduction

In control systems, the main task of the controller is to obtain an admissible control law when certain conditions are satisfied according to the dynamic characteristics of the plant. Then, the plant cooperates with other controllers to achieve the optimization of performance indicators (maximum or minimum), so as to solve the optimal control problem [1]. Solving the optimal control problem is equivalent to solving the Hamilton–Jacobi–Bellman (HJB) equation. Because of the nonlinearity and partial derivative of the HJB equation, it is difficult to acquire its analytical solution. In recent years, the adaptive dynamic programming (ADP) method has received widespread attention for obtaining the approximate solutions of the HJB equation [2,3,4,5]. The ADP method combines the idea of dynamic programming (DP) and reinforcement learning (RL). The RL-based agent interacts with the environment and takes action to obtain cumulative rewards, and it overcomes the problem of dimensional disasters generated in DP. The implementation of the ADP method approximates the HJB equation by the approximation principle of neural network function, and obtains the optimal value function and optimal control law by RL. In [6], the structure of ADP is proposed for the first time. Subsequently, in [7] uses the actor–critic NN structure to deal with the nonlinear system that depends on the dynamic information. In [8,9,10], the policy iteration (PI) consisting of policy evaluation and policy improvement iterative techniques is used for continuous time and discrete time dynamic systems to iteratively update the control policy online with state and input information. However, the PI learning algorithm relies on a precise dynamic system. Both drift dynamic and input dynamic are needed in the iterative process. Therefore, it is based on the improvement of the PI algorithm and uses the iterative technology of PI approximation to the optimal solution. An integral reinforcement learning (IRL) algorithm in [11] is proposed to release drift dynamics by adding integral operations. In [12], for a locally unknown continuous-time nonlinear system with actuator saturation, an IRL algorithm with an actor-critic network is proposed to solve the HJB equation online. Then, the experience replay technique is used to update the critic weights to solve the IRL-Bellman equation. The IRL method is used for the unknown $H_{\infty }$ tracking control problem of a linear system with actuator saturation in [13]. The designed controller not only makes the system state trajectory of convergence but the control strategy can also make the tracking error asymptotically stable.

The complexity and uncertainty of the system increase the difficulty of solving the HJB solution. Especially the nonlinear system in practical problems, $H_{\infty }$ control problem is an alternative method to solve the robust optimal problem. The purpose of the $H_{\infty }$ control problems is to design a controller that can effectively suppress the impact of external disturbances on system performance. Therefore, many studies propose the robust optimal control method to transform the $H_{\infty }$ optimal control problem into the zero-sum game problem, which is a max/min optimization problem essentially [14,15]. Therefore, the Nash saddle point of the two-player zero-sum problem is regarded as solving the solution of the Hamilton–Jacobi–Isaacs (HJI) equation. In [16,17], an online policy iteration (PI) algorithm is presented to solve the two-player zero-sum game. An ADP-based critic–actor network is used to approximate the solution of the HJI equation. For the system containing external disturbances, a new ADP-based online IRL algorithm is proposed to approximate the HJI equation. Use current data and historical data to update network weights to improve data utilization efficiency when solving the HJI equation [18]. In [19], an $H_{\infty }$ tracking controller is designed to solve the zero-sum game with completely unknown dynamics and constrained input. The tracking HJI equation is solved by off-policy reinforcement learning. In [20,21], Adaptive critic designs (ACD) are used to solve the zero-sum problem that external disturbances have an impact on system performance.

However, in engineering applications, the controller requires high physical characteristics and security, which is usually limited by the threshold. In addition, the design and analyzable methods of the control system also have higher requirements. It is not only necessary to achieve the control design goal to ensure the stability of the dynamic system but also to consider the control performance of the system to save energy and reduce consumption. The above iterative methods mostly use the time-triggered mechanism of periodic sampling to solve the nonlinear zero-sum game problem with actuator saturation.

To reduce unnecessary data transmission and frequency update between components, the event-triggered mechanism is introduced into adaptive dynamic programming for the first time in [22]. A controller with sampling states limited by a triggering condition is designed, which not only ensures the stability and optimality of the system, but also reduces the controller update. For the system with constrained input, [23,24] propose an approximately optimal control structure based on the event-triggered strategy, which makes control laws update non-periodically to reduce computation and transmission costs, and ensures the uniform ultimate boundedness of the event-triggered system. The designed triggering condition is the key to the event-based controller. Not only does the triggering condition have a non-negative triggering threshold but it avoids Zeno behavior. In [25], an event-driven controller is designed, and the triggering condition should not only have a non-negative triggering threshold but also the Zeno behavior is free. In [26], the IRL-based event-triggered algorithm is used to solve the partially unknown nonlinear system. The critic-network updates periodically and the actor-network updates non-periodically to approximately acquire the performance index function and control law. The good convergence of the NN weight is theoretically proved and effectively avoids the Zeno phenomenon. For the system with unknown disturbance, the robust controller is designed by using the $H_{\infty }$ control method. In [15], an event-triggered $H_{\infty }$ controller is structured, which introduces the disturbance attenuation level into the triggering condition, and ensures that the triggering threshold is non-negative by selecting appropriate parameters. The control law is updated with an event-triggered strategy, and the perturbation law is adjusted under the time-triggered mechanism.

At present, the event-triggered ADP algorithm has been applied to optimal regulation problem [27,28], optimal tracking control [29], zero-sum game [30,31,32], non-zero-sum game [33,34], robust control problem [35,36]. However, most studies are based on identification–critic NNs or critic–actor–perturbation NNs structure to approximate obtain the solution of the HJI equation for the zero-sum game problem, which often increases the communication load such as actuators and controllers, and increases the loss of resources and costs [37,38]. Therefore, an event-triggered ADP method is proposed to solve the $H_{\infty }$ optimal control problem for partially unknown continuous-time nonlinear systems with constrained input, and the structure of a single-critic network is constructed to acquire approximately the solution of the HJI equation. This paper aims to achieve the following aspects:

• Based on the event-triggered control, the triggering condition with the level of disturbance attenuation is developed to limit the sampling states of the system, and the appropriate level of disturbance attenuation is selected to ensure that the triggering condition remains non-negative.
• An event-triggered $H_{\infty }$ input-constrained nonlinear system controller is designed. The event-based control law and disturbance law are updated at the triggering instant, and the computation in the control process is effectively reduced.
• For the zero-sum game problem, a single-critic network structure based on the event-triggered ADP is proposed to approximate the solution of the HJI equation. It not only greatly reduces the update frequency of the controller and reduces the computational cost, but also the reliance on known dynamic information is relaxed.

The rest of this article is organized as follows. Section 2 gives the description and transformation of the problem. Section 3 introduces the event-triggered HJI equation. Section 4 describes the implementation of the event-based ADP algorithm, and gives an analysis of system stability. Section 5 demonstrates the simulation of the continuous-time linear system and the continuous-time (CT) nonlinear system, and Section 6 presents the conclusion of this paper.

Notation. Some parameters are defined in this article. $\mathbb{R}$, ${\mathbb{R}}^m$, ${\mathbb{R}}^{m\times n}$ are all denoted the set of real matrices, and m, $m\times n$ are represented as the corresponding dimension matrix. ${\mathbb{N}}^{+}$ is the set of positive real numbers. ${tanh}^{-\mathsf{T}}(·)$ is expressed as an inverse hyperbolic function, $\underline{\lambda }(·)$ and $\overline{\lambda }(·)$ are defined as the minimum and maximum eigenvalue of the matrix. $\parallel ·\parallel$ express as the 2-norm. $\nabla V\equiv \partial V/\partial x$ is denoted as the partial derivative of the function V with respect to the variable x, → denotes as numerical infinite approximation.

2. Problem Description

Consider the continuous-time nonlinear system with external disturbance as

$$\dot{x}=f\left(x\left(t\right)\right)+g\left(x\left(t\right)\right)u\left(t\right)+h\left(x\left(t\right)\right)\upsilon \left(t\right)$$

(1)

where $x\in {\mathbb{R}}^n$ denotes the states vector of system. $u\left(t\right)=\{(u_1,u_2,\cdots ,u_m)\in {\mathbb{R}}^m:\parallel u_i\parallel \le u_M,i=1,2,\cdots ,m\}$, $u_M$ is a positive upper bound. $f\left(x\right)\in {\mathbb{R}}^n$ is drift dynamic, and has $f\left(0\right)=0$. $g\left(x\right)\in {\mathbb{R}}^{n\times m}$ is input coupling dynamic. $h\left(x\right)\in {\mathbb{R}}^{n\times q}$ and $\upsilon \left(t\right)\in {\mathbb{R}}^q$ are the disturbance dynamics and bounded external disturbance. ${\upsilon }_M$ is a positive constant with $\parallel \upsilon \left(t\right)\parallel \le {\upsilon }_M$.

Since the system is affected by constrained input, $M\left(u\right)$ is defined as [39]

$$\begin{array}{c}\hfill M\left(u\right)=2{\int }_0^u{u}_M{tanh}^{-\mathsf{T}}(\mu /u_M)Rd\mu \\ \hfill =2\sum _{i=1}^m{\int }_0^{u_i}{u}_M{tanh}^{-\mathsf{T}}({\mu }_i/u_M)Rd{\mu }_i\end{array}$$

(2)

where $M\left(u\right)$ represents nonquadratic function, and R is assumed to be diagonal matrix. ${tanh}^{-\mathsf{T}}(·)$ is a hyperbolic tangent function, which is treated as the constrained input.

For the $H_{\infty }$ control problem with constrained input, we need to reduce the impact of external disturbances on system performance. For any $\upsilon \left(t\right)\in L_2[0,\infty )$, the closed-loop system (1) exists

$${\int }_t^{\infty }\left(x^{\mathsf{T}}Qx+M\left(u\right)\right)d\tau \le {\gamma }^2{\int }_t^{\infty }{\upsilon }^{\mathsf{T}}\left(\tau \right)\upsilon \left(\tau \right)d\tau$$

(3)

and this is, $L_2$-gain not larger than $\gamma$ is satisfied, where $Q\in {\mathbb{R}}^{n\times n}$ is a symmetric positive matrix. $\gamma >0$ denotes the level of disturbance attenuation.

The design intention of the $H_{\infty }$ optimal control problem is to find a control law that not only guarantees the asymptotic stability of the system (1) but also the disturbance attenuation condition (3) holds. Then, we need to define the following value function

$$J(x\left(t\right),u,v)={\int }_t^{\infty }U\left(x\left(\tau \right),u\left(\tau \right),\upsilon \left(\tau \right)\right)d\tau$$

(4)

where $U(x,u,\upsilon )=x^{\mathsf{T}}Qx+M\left(u\right)-{\gamma }^2{\upsilon }^{\mathsf{T}}\upsilon$.

To achieve the above goals, the $H_{\infty }$ control problem is treated as the zero-sum game problem. By the minimax optimization principle, the perturbation policy is used as a decision-maker and maximizes its value, while the control policy acts as another decision-maker and minimizes its value. The following optimal value function is indicated as

$$V^{*}\left(x\right)=J(x\left(t\right),u^{*},v^{*})=\underset{u}{min}\underset{\upsilon }{max}{\int }_t^{\infty }U\left(x\left(\tau \right),u\left(\tau \right),\upsilon \left(\tau \right)\right)d\tau$$

(5)

Assume that the $V\left(x\right)=J(x,u,\upsilon )$ is continuous and differentiable, and the Bellman equation can be given as

$$\nabla V^{\mathsf{T}}(f+gu+h\upsilon )+x^{\mathsf{T}}Qx+M\left(u\right)-{\gamma }^2{\upsilon }^{\mathsf{T}}\upsilon =0$$

(6)

Then, the Hamiltonian function is defined as

$$H(x,\nabla V,u,\upsilon )=\nabla V^{*\mathsf{T}}(f+gu+h\upsilon )+M\left(u\right)+x^{\mathsf{T}}Qx-{\gamma }^2{\upsilon }^{\mathsf{T}}\upsilon$$

(7)

Definition 1

([12]). A control law is defined as admission with respect to the value function (4) on a compact set Λ. If $u\left(0\right)=0$ and $\upsilon \left(0\right)=0$, $u\left(x\right)$ can ensure system (1) stability on Λ and for any $x_0\in \Lambda$ the value funcion $V\left(x_0\right)$ is finite.

By stationarity conditions $\partial H/\partial u=0$ and $\partial H/\partial \upsilon =0$, the optimal control law $u^{*}\left(x\right)$ and the disturbance law ${\upsilon }^{*}\left(x\right)$ are expressed as

$$u^{*}\left(x\right)=-u_{M}tanh\left(\frac{1}{2u_M}{R}^{-1}{g}^{\mathsf{T}}\left(x\right)\nabla V^{*}\left(x\right)\right)$$

(8)

$${\upsilon }^{*}\left(x\right)=\frac{1}{2}{\gamma }^{-2}{h}^{\mathsf{T}}\left(x\right)\nabla V^{*}\left(x\right)$$

(9)

Submitting (8) into (2), $M\left(u^{*}\right)$ can be obtained

$$M\left(u^{*}\right)=u_M\nabla {V^{*}}^{\mathsf{T}}gtanh(\Psi )+u_M^2\overline{R}ln\left(\underline{1}-{tanh}^2(\Psi )\right)$$

(10)

where $\Psi =1/\left(2u_M\right)R^{-1}{g}^{\mathsf{T}}\nabla V^{*}\left(x\right)$, and $\underline{1}$ indicates the elements are all 1 column vectors, $\overline{R}$ is a row vector made up of the diagonal elements of R.

Based on (7)–(9), the optimal time-triggered HJI equation becomes

$$\begin{array}{c}\hfill \nabla V^{*\mathsf{T}}\left(x\right)f\left(x\right)+x^{\mathsf{T}}Qx+u_M^2\overline{R}ln\left(\underline{1}-{tanh}^2(\Psi )\right)-\frac{1}{4}{\gamma }^{-2}\nabla V^{*\mathsf{T}}h\left(x\right)h^{\mathsf{T}}\left(x\right)\nabla V^{*}\left(x\right)=0\end{array}$$

(11)

From (11), it is difficult to solve its solution. In addition, since the dynamic system $f\left(x\right)$ is unknown, it further increases the difficulty of solving the HJI equation. Thus, we prefer to introduce the IRL technology to obtain the solution to the HJI equation without requiring the system dynamic $f\left(x\right)$.

The learning process of IRL mainly includes the following steps in Algorithm 1.

Algorithm 1: IRL algorithm.

1: Choose the initial admissible control laws $u_0$, ${\upsilon }_0$. $\mathbf{2}:\mathbf{Policy}\mathbf{Evaluation}.$ During the interval time $[t-T,t]$, the value function $V^{\left[i\right]}\left(x\right)$ is obtained by solving the following Bellman equation $$V^{\left[i\right]}\left(x(t-T)\right)={\int }_{t-T}^t\left(x^{\mathsf{T}}Qx+M\left(u^{\left[i\right]}\right)-{\gamma }^2{\upsilon }^{\left[i\right]\mathsf{T}}{\upsilon }^{\left[i\right]}\right)d\tau +V^{\left[i\right]}\left(x\left(t\right)\right)$$ (12) $\mathbf{3}:\mathbf{Policy}\mathbf{Inprovement}.$ Update the control law and disturbance law by $$u^{[i+1]}\left(x\right)=-u_{M}tanh\left(\frac{1}{2u_M}{R}^{-1}{g}^{\mathsf{T}}\left(x\right)\nabla V^{\left[i\right]}\left(x\right)\right)$$ (13) $${\upsilon }^{[i+1]}\left(x\right)=\frac{1}{2}{\gamma }^{-2}{h}^{\mathsf{T}}\left(x\right)\nabla V^{\left[i\right]}\left(x\right)$$ (14)

Remark 1.

The IRL algorithm is an improvement over the policy iterative (PI) technique. Compared with the standard Bellman Equation (4), the IRL-based Bellman Equation (12) does not involve system dynamics $f\left(x\right)$ and $g\left(x\right)$.

However, in the IRL algorithm, the control policy is updated at t and applied to the next interval $[t,t+T]$ of the system. It is so-called periodic sampling or time-triggered control (TTC). By the TTC, the control law and disturbance law are periodically updated, which may cause large numerous resource consumption and computation costs for the network system with limited bandwidth. Therefore, an event-triggered ADP is applied to solve the HJI equation for the $H_{\infty }$ control problem.

3. Design of Event-Triggered ${\mathit{H}}_{\infty }$-Constrained Optimal Control

An event-triggered control is introduced into ADP to solve the zero-sum game problem and then develops an event-triggered HJI equation. Then, a triggering condition is designed to limit the number of sampling states which can avoid the Zeno behavior.

In the ETC, the event-triggered instant sequences are defined as ${\left\{t_k\right\}}_{k=0}^{\infty }$, $t_k<t_{k+1}$ and $k\in {\mathbb{N}}^{+}$. A novel sampling state can be obtained at the triggering instant when the triggering condition is violated. The event error determines the appearance of the sampling state, and it can be expressed as

$$e_k\left(t\right)={\widehat{x}}_k-x\left(t\right),t\in [t_k,t_{k+1})$$

(15)

where ${\widehat{x}}_k=x\left(t_k\right)$ is the sampling state at the triggering instant $t_k$ and $x\left(t\right)$ represents the current state, respectively.

Remark 2.

The event is triggered depending on two basic conditions, which are the event-triggered error $e_k$ and the triggering threshold $e_T$. An event occurs and the controller updates when the error violates the triggering threshold, this is, the error is $e_k=0$ at $t=t_k$. The holding of the sampling state in the non-triggered interval $t\in [t_k,t_{k+1})$ is realized by the zero-order holder (ZOH). The ZOH is used to store the control law of the previous moment and convert the sampled signal into a continuous signal. Then, the law is kept until the next triggering instant.

It is assumed that there is no delay between the sensor and the controller. According to (15), the closed-loop system (1) is rewritten as

$$\dot{x}=f\left(x\right)+g\left(x\right)u\left({\widehat{x}}_k\right)+h\left(x\right)\upsilon \left({\widehat{x}}_k\right),t\in [t_k,t_{k+1})$$

(16)

Then, the event-based value function is shown as

$$\begin{array}{c}\hfill V\left(x(t-T)\right)={\int }_{t-T}^t\left(x^{\mathsf{T}}Qx+M\left(u\left({\widehat{x}}_k\right)\right)-{\gamma }^2{\upsilon }^{\mathsf{T}}\left({\widehat{x}}_k\right)\upsilon \left({\widehat{x}}_k\right)\right)d\tau +V\left(x\left(t\right)\right)\end{array}$$

(17)

Therefore, the optimal event-triggered control law and the disturbance law in $t\in [t_k,t_{k+1})$ can be transformed into

$$u^{*}\left({\widehat{x}}_k\right)=-u_{M}tanh\left(\frac{1}{2u_M}{R}^{-1}{g}^{\mathsf{T}}\left({\widehat{x}}_k\right)\nabla V^{*}\left({\widehat{x}}_k\right)\right)$$

(18)

$${\upsilon }^{*}\left({\widehat{x}}_k\right)=\frac{1}{2}{\gamma }^{-2}{h}^{\mathsf{T}}\left({\widehat{x}}_k\right)\nabla V^{*}\left({\widehat{x}}_k\right)$$

(19)

Similar to (11), the event-triggered HJI equation becomes

$$\begin{array}{cc}\hfill \nabla V& {}^{*\mathsf{T}}\left(x\right)f\left(x\right)+x^{\mathsf{T}}Qx+M\left(u^{*}\left({\widehat{x}}_k\right)\right)-u_M{V}^{*\mathsf{T}}\left(x\right)gtanh\left(\frac{1}{2u_M}{R}^{-1}{g}^{\mathsf{T}}\left({\widehat{x}}_k\right)\nabla V^{*}\left({\widehat{x}}_k\right)\right)\hfill \\ \hfill & +\frac{1}{2}{\gamma }^{-2}\nabla V^{*\mathsf{T}}\left(x\right)h\left(x\right)h^{\mathsf{T}}\left({\widehat{x}}_k\right)\nabla V^{*}\left({\widehat{x}}_k\right)-\frac{1}{4}{\gamma }^{-2}\nabla V^{*\mathsf{T}}\left({\widehat{x}}_k\right)h\left({\widehat{x}}_k\right)h^{\mathsf{T}}\left({\widehat{x}}_k\right)\nabla V^{*}\left({\widehat{x}}_k\right)=0\hfill \end{array}$$

(20)

To sum up, in order to reduce communication load and computational cost, we introduce the event-triggered ADP in Algorithm 2 as follows

Algorithm 2 Event-triggered ADP algorithm.

$\mathbf{1}:$ Choose the initial admissible control laws $u_0$, ${\upsilon }_0$. $\mathbf{2}:\mathbf{Policy}\mathbf{Evaluation}.$ During the interval time, the value function $V\left(x\right)$ can be obtain by solving the event-based Bellman equation (17). $\mathbf{3}:\mathbf{Policy}\mathbf{Inprovement}.$ (18) and (19) can obtain the optimal control policy $u^{*}\left({\widehat{x}}_k\right)$ and the optimal disturbance policy ${\upsilon }^{*}\left({\widehat{x}}_k\right)$ separately.

Assumption 1

([15]). For the optimal value function $V^{*}\left(x\right)$ is continuously differentiable, $V^{*}\left(x\right)$ and $\nabla V^{*}\left(x\right)$ are bounded on Λ, such that the solution $V^{*}\left(x\right)$ and its partial derivative $\nabla V^{*}\left(x\right)$ can be denoted as $max\{\parallel V^{*}\left(x\right)\parallel ,\parallel \nabla V^{*}\left(x\right)\parallel \}\le {\eta }_0$ with a positive constant ${\eta }_0$.

Assumption 2

([26]). For $\forall x\in {\mathbb{R}}^n$, there exist $g_M>0$ and $h_M>0$ and can be satisfied with $\parallel g\left(x\right)\parallel \le g_M$ and $\parallel h\left(x\right)\parallel \le h_M$.

Assumption 3

Assuming that the optimal control law and the disturbance law are Lipschitz continuous on a compact set $\Lambda \in {\mathbb{R}}^n$. There exist $L_u>0$ and $L_{\upsilon }>0$ satisfying

$$∥u^{*}\left(x\right)-u^{*}\left({\widehat{x}}_k\right)∥\le L_u∥e_k\left(t\right)∥$$

(21)

$$∥{\upsilon }^{*}\left(x\right)-{\upsilon }^{*}\left({\widehat{x}}_k\right)∥\le L_{\upsilon }∥e_k\left(t\right)∥$$

(22)

Theorem 1.

Let $V^{*}\left(x\right)$ be the optimal solution of (11). The optimal control laws $u^{*}\left({\widehat{x}}_k\right)$ in (18) and ${\upsilon }^{*}\left(x\right)$ in (19) are obtained to ensure that the dynamic system is uniformly ultimately bounded (UUB) only when the following triggering condition is given as

$$∥e_k\left(t\right)∥\le \sqrt{\frac{(1-{\sigma }^2)}{L_u^2+{\gamma }^2{L}_{\upsilon }^2}\underline{\lambda }\left(Q\right){∥x∥}^2}\triangleq e_T$$

(23)

where $\gamma >0$ is the level of disturbance attenuation from (3), σ is a positive design parameter. $\underline{\lambda }\left(Q\right)$ is the minimum eigenvalue of the matrix Q. The controller gets a new control law when the event error $e_k$ exceeds the triggering threshold $e_T$.

Proof. 

The optimal value function $V^{*}\left(x\right)$ is obtained from (11) and it is positive definite for any $x\ne 0$. Use $V^{*}\left(x\right)$ as the Lyapunov function, and its derivative to time is defined as

$$\begin{array}{cc}\hfill {\dot{V}}^{*}\left(x\right)& =\nabla V^{*\mathsf{T}}\left(x\right)\left(f+g{u}^{*}\left({\widehat{x}}_k\right)+h{\upsilon }^{*}\left({\widehat{x}}_k\right)\right)\hfill \\ \hfill & =\nabla V^{*\mathsf{T}}\left(x\right)g\left(u^{*}\left({\widehat{x}}_k\right)-u^{*}\left(x\right)\right)+\nabla V^{*\mathsf{T}}\left(x\right)h\left({\upsilon }^{*}\left({\widehat{x}}_k\right)-{\upsilon }^{*}\left(x\right)\right)\hfill \\ \hfill & +\nabla V^{*\mathsf{T}}\left(x\right)\left(f+g{u}^{*}\left(x\right)+h{\upsilon }^{*}\left(x\right)\right)\hfill \end{array}$$

(24)

From (6), we can acquire

$$\nabla V^{*\mathsf{T}}\left(x\right)\left(f+g{u}^{*}\left(x\right)+h{\upsilon }^{*}\left(x\right)\right)=-x^{\mathsf{T}}Qx-M\left(u^{*}\left(x\right)\right)+{\gamma }^2{\upsilon }^{*\mathsf{T}}{\upsilon }^{*}$$

(25)

Then, (8) and (9) can be transformed as

$$\nabla V^{*\mathsf{T}}\left(x\right)g=-2u_{M}R{tanh}^{-\mathsf{T}}\left(\frac{u^{*}\left(x\right)}{u_M}\right)$$

(26)

$$\nabla V^{*\mathsf{T}}\left(x\right)h=2{\gamma }^2{{\upsilon }^{*}}^{\mathsf{T}}\left(x\right)$$

(27)

According to (25)–(27), (24) can be rewritten as

$$\begin{array}{cc}\hfill {\dot{V}}^{*}\left(x\right)& =\underset{s_1}{\underbrace{2u_{M}R{tanh}^{-\mathsf{T}}\left(\frac{u^{*}\left(x\right)}{u_M}\right)\left(u^{*}\left(x\right)-u^{*}\left({\widehat{x}}_k\right)\right)}}-x^{\mathsf{T}}Qx\hfill \\ \hfill & +\underset{s_2}{\underbrace{2{\gamma }^2{\upsilon }^{*\mathsf{T}}\left(x\right)\left({\upsilon }^{*}\left({\widehat{x}}_k\right)-{\upsilon }^{*}\left(x\right)\right)}}-M\left(u^{*}\left(x\right)\right)+{\gamma }^2{{\upsilon }^{*}}^{\mathsf{T}}{\upsilon }^{*}\hfill \end{array}$$

(28)

By the weighted form of Young’s inequality, we have

$$\begin{array}{cc}\hfill s_1& \le \parallel u_{M}R{tanh}^{-\mathsf{T}}\left(\frac{u^{*}\left(x\right)}{u_M}\right){\parallel }^2+{∥u^{*}\left(x\right)-u^{*}\left({\widehat{x}}_k\right)∥}^2\hfill \\ \hfill & \le \frac{1}{4}\parallel \nabla V^{*\mathsf{T}}\left(x\right)g{\parallel }^2+L_u^2{∥e_k\left(t\right)∥}^2\hfill \end{array}$$

(29)

Similarly, we can obtain

$$\begin{array}{cc}\hfill s_2& \le {\gamma }^2{\parallel {\upsilon }^{*}\parallel }^2+{\gamma }^2{∥{\upsilon }^{*}\left({\widehat{x}}_k\right)-{\upsilon }^{*}\left(x\right)∥}^2\hfill \\ \hfill & \le {\gamma }^2{\parallel {\upsilon }^{*}\parallel }^2+{\gamma }^2{L}_{\upsilon }^2{∥e_k\left(t\right)∥}^2\hfill \end{array}$$

(30)

Let Assumption 1–3 holding and note $x^{\mathsf{T}}Qx\ge \underline{\lambda }\left(Q\right){∥x∥}^2$, we have

$$\begin{array}{cc}\hfill {\dot{V}}^{*}\left(x\right)& =-x^{\mathsf{T}}Qx-M\left(u^{*}\left(x\right)\right)+2{\gamma }^2{\parallel {\upsilon }^{*}\left(x\right)\parallel }^2+{\gamma }^2{L}_{\upsilon }^2{∥e_k\left(t\right)∥}^2\hfill \\ \hfill & +\frac{1}{4}\parallel \nabla V^{*\mathsf{T}}\left(x\right)g{\parallel }^2+L_u^2{∥e_k\left(t\right)∥}^2\hfill \\ \hfill & \le -(1-{\sigma }^2)\underline{\lambda }\left(Q\right){∥x∥}^2-{\sigma }^2\underline{\lambda }\left(Q\right){∥x∥}^2+(L_u^2+{\gamma }^2{L}_{\upsilon }^2){\parallel e_k\parallel }^2\hfill \\ \hfill & +\frac{1}{4}\parallel \nabla V^{*\mathsf{T}}\left(x\right)g{\parallel }^2+\frac{1}{2{\gamma }^2}\parallel \nabla V^{*\mathsf{T}}\left(x\right)h{\parallel }^2-M\left(u^{*}\left(x\right)\right)\hfill \\ \hfill & \le -(1-{\sigma }^2)\underline{\lambda }\left(Q\right){∥x∥}^2-M\left(u^{*}\left(x\right)\right)+\left(L_u^2+{\gamma }^2{L}_{\upsilon }^2\right){∥e_k∥}^2-{\sigma }^2\underline{\lambda }\left(Q\right){∥x∥}^2+{\omega }_0\hfill \end{array}$$

(31)

where ${\omega }_0=(1/4){\eta }_0^2{g}_M^2+\left({\eta }_0^2{h}_M^2\right)/\left(2{\gamma }^2\right)$. From (10), we can obtain $-M\left(u^{*}\right)\le 0$. If the triggering condition is given by (23), we can obtain

$${\dot{V}}^{*}\left(x\right)\le -{\sigma }^2\underline{\lambda }\left(Q\right){∥x∥}^2+{\omega }_0$$

(32)

Thus, it can conclude that ${\dot{V}}^{*}\left(x\right)<0$ for $\parallel x\parallel >\sqrt{{\omega }_0/\left({\sigma }^2\underline{\lambda }\left(Q\right)\right)}$. Therefore, the optimal laws $u^{*}\left({\widehat{x}}_k\right)$ and ${\upsilon }^{*}\left({\widehat{x}}_k\right)$ can ensure that the closed-loop system (1) is asymptotically stable by the Lyapunov theory. The proof is completed. □

From (23), we can acquire the triggering interval currently when the event is triggered. The difference between the two triggering instants is recorded as the triggering interval. The interval can be expressed as ${\tau }_k=t_{k+1}-t_k$. However, the Zeno behavior occurs if there exists ${\left({\tau }_k\right)}_{min}=0$.

Remark 3.

Zeno behavior is a special dynamic behavior in hybrid systems in which an infinite number of discrete transitions occur in a finite amount of time [40]. The existence of the Zeno behavior does not guarantee that the system can be asymptotically stable. Therefore, it is required to avoid Zeno behavior.

Theorem 2.

Under the triggering conditions (23) given in Theorem 1, the inter-execution time exists ${\tau }_k=t_{k+1}-t_k$ and satisfies

$${\tau }_k\ge \frac{\Gamma \left(t\right)}{L_x\left(1+\Gamma \left(t\right)\right)}$$

(33)

where $\Gamma \left(t\right)=\parallel e_k\left(t\right)\parallel /\parallel x\left(t\right)\parallel$, then the minimum interval time is not less than the positive constant.

Proof. 

Based on [41,42], for $t\in (t_k,t_{k+1})$, we have

$$\begin{array}{cc}\hfill \dot{\Gamma }\left(t\right)& =\frac{d}{dt}\frac{\parallel e_k\parallel }{∥x∥}=\frac{d}{dt}\frac{{\left(e_k^{\mathsf{T}}{e}_k\right)}^{\frac{1}{2}}}{{\left(x^{\mathsf{T}}x\right)}^{\frac{1}{2}}}\hfill \\ \hfill & =\frac{{\left(e_k^{\mathsf{T}}{e}_k\right)}^{-\frac{1}{2}}{e}_k^{\mathsf{T}}{\dot{e}}_k{\left(x^{\mathsf{T}}x\right)}^{\frac{1}{2}}}{{∥x∥}^2}-\frac{{\left(x^{\mathsf{T}}x\right)}^{-\frac{1}{2}}{x}^{\mathsf{T}}\dot{x}{\left(e_k^{\mathsf{T}}{e}_k\right)}^{\frac{1}{2}}}{{∥x∥}^2}\hfill \\ \hfill & =-\frac{e_k^{\mathsf{T}}\dot{x}}{\parallel e_k\parallel \parallel x\parallel }-\frac{x^{\mathsf{T}}\dot{x}\parallel e_k\parallel }{{\parallel x\parallel }^3}\hfill \end{array}$$

(34)

According to (15) and its gradient is ${\dot{e}}_k\left(t\right)={\dot{\widehat{x}}}_k-\dot{x}\left(t\right)$. Since ${\widehat{x}}_k$ is a constant for $t\in [t_k,t_{k+1})$, we obtain ${\dot{e}}_k\left(t\right)=-\dot{x}\left(t\right)$. For the drift dynamid $f\left(x\right)$ is Lipschitz continuous, there exists $L_x>0$ and satisfies $\parallel f\left(x\right)\parallel \le L_x\parallel x\parallel$, such that the following inequality holds [36]

$$\begin{array}{cc}\hfill ∥\dot{x}∥& =∥{\dot{e}}_k∥=\parallel f\left(x\right)+g\left(x\right)u\left({\widehat{x}}_k\right)+h\left(x\right)\upsilon \left({\widehat{x}}_k\right)\parallel \hfill \\ \hfill & \le \parallel f\left(x\right)\parallel +\parallel g\left(x\right)u\left({\widehat{x}}_k\right)\parallel +\parallel h\left(x\right)\upsilon \left({\widehat{x}}_k\right)\parallel \hfill \\ \hfill & \le L_x\parallel x+e_k\parallel \hfill \end{array}$$

(35)

Thus, we can obtain

$$\begin{array}{cc}\hfill \dot{\Gamma }\left(t\right)& \le \frac{\parallel e_k\parallel \parallel \dot{x}\parallel }{\parallel e_k\parallel \parallel x\parallel }+\frac{∥x∥∥\dot{x}∥}{\parallel x\parallel \parallel x\parallel }\frac{\parallel e_k\parallel }{∥x∥}\hfill \\ \hfill & =\frac{∥\dot{x}∥}{\parallel x\parallel }\left(1+\frac{\parallel e_k\parallel }{\parallel x\parallel }\right)\hfill \\ \hfill & \le \frac{L_x∥e_k∥+L_x∥x∥}{∥x∥}(1+\frac{\parallel e_k\parallel }{∥x∥})\hfill \\ \hfill & =L_x{\left(1+\frac{\parallel e_k\parallel }{\parallel x\parallel }\right)}^2\hfill \end{array}$$

(36)

From (33) and (36) can be rewritten as $\dot{\Gamma }\left(t\right)\le L_x(1+\Gamma (t){)}^2$. Let $\varrho \left(t\right)$ be the solution to $\dot{\varrho }\left(t\right)=L_x(1+\varrho (t){)}^2$ and satisfying $\varrho \left(0\right)$=0. Then, its solution can be expressed as $\varrho \left(t\right)=L_x(t-t_k)/\left(1-L_x(t-t_k)\right)$ where $t\ge t_k$. Therefore, we obtain $\Gamma \left(t\right)\le \varrho \left(t\right)$ by comparison principle. Assume that the instant time $\xi \in {\mathbb{R}}^{+}$, and have $t_k\le \xi \le t_{k+1}$, we can obtain $\Gamma \left(\xi \right)\le \varrho \left(\xi \right)$. Then, we obtain $\xi \ge \eta +t_k$ with $\eta =\Gamma \left(t\right)/\left(L_x(1+\Gamma \left(t\right))\right)$. Since $\eta +t_k\le \xi \le t_{k+1}$, we have ${\tau }_k=t_{k+1}-t_k\ge \eta$. For the asymptotically stable closed-loop system, we can obtain $e_k\left(t\right)>0$ for any $\parallel x\left(t\right)\parallel \ne 0$. Therefore, $\Gamma \left(t\right)>0$ is held, which is guaranteed to have a lower bound with ${\left({\tau }_k\right)}_{min}\ge \eta >0$. The proof is accomplished. □

4. Approximate Solution of Event-Triggered HJI Equation

An ADP-based event-triggered algorithm is shown before. In this section, an event-based single neural network with the approximation function is constructed, namely critic NN, to approximately solve the HJI equation.

Based on the general approximation of neural networks. The value function $V\left(x\right)$ and its gradient $\nabla V\left(x\right)$ can be obtained by the neural network as

$$V\left(x\right)=W_c^{\mathsf{T}}\varphi \left(x\right)+{\epsilon }_V\left(x\right)$$

(37)

$$\nabla V\left(x\right)=\nabla \varphi {\left(x\right)}^{\mathsf{T}}{W}_c+\nabla {\epsilon }_V\left(x\right)$$

(38)

where $W_c\in {\mathbb{R}}^{L_1}$ is the ideal weight the critic NN, $L_1$ is the number of neurons, $\varphi \left(x\right)\in {\mathbb{R}}^{L_1}$ is the suitable activation function of NN. ${\epsilon }_V\left(x\right)$ indicates the reconstruction error.

The approximate value function can be obtained from (37), and substituted it into (12) to obtain the following Bellman equation

$${\int }_{t-T}^t\left(x^{\mathsf{T}}Qx+M\left(u\right)-{\gamma }^2{\upsilon }^{\mathsf{T}}\upsilon \right)d\tau +W_c^{\mathsf{T}}\Delta \varphi \left(x\right)\triangleq e_B\left(t\right)$$

(39)

where $\Delta \varphi \left(x\right)=\varphi \left(x\right(t\left)\right)-\varphi \left(x\right(t-T\left)\right)$. Due to the existence of the NN approximation error, the Bellman equation error is

$${\epsilon }_B\left(t\right)={\epsilon }_V\left(t\right)-{\epsilon }_V(t-T)=-{\int }_{t-T}^t\left(\nabla {\epsilon }_V^{\mathsf{T}}(f\left(x\right)+g\left(x\right)u+h\left(x\right)\upsilon )\right)d\tau$$

(40)

For (39), the optimal value is obtained only when $e_B$ goes to zero indefinitely. However, the ideal weights $W_c$ are unknown, they can be estimated by the current weights ${\widehat{W}}_c$. The actual value function of the approximate network can be expressed as

$$\widehat{V}\left(x\right)={\widehat{W}}_c^{\mathsf{T}}\varphi \left(x\right)$$

(41)

$$\nabla \widehat{V}\left(x\right)=\nabla \varphi {\left(x\right)}^{\mathsf{T}}{\widehat{W}}_{\mathrm{c}}$$

(42)

To solve the optimal control problem, the value function is obtained by solving the Bellman equation in the policy evaluation, and then the new control laws are obtained by the present value function in the policy improvement. Combined with (20), (21), and (42), the event-triggered control law and disturbance law become

$$\widehat{u}\left({\widehat{x}}_k\right)=-u_{M}tanh\left(\frac{1}{2u_M}{R}^{-1}{g}^{\mathsf{T}}\left({\widehat{x}}_k\right)\nabla {\varphi }^{\mathsf{T}}\left({\widehat{x}}_k\right){\widehat{W}}_c\right)$$

(43)

$$\widehat{\upsilon }\left({\widehat{x}}_k\right)=\frac{1}{2}{\gamma }^{-2}{h}^{\mathsf{T}}\left({\widehat{x}}_k\right)\nabla {\varphi }^{\mathsf{T}}\left({\widehat{x}}_k\right){\widehat{W}}_c$$

(44)

Combining (12) with (41), the approximate Bellman equation is described as

$$\widehat{q}\left(t\right)+{\widehat{W}}_c^{\mathsf{T}}\Delta \varphi \left(x\right)\triangleq e_c$$

(45)

where the reinforcement signal during the integral interval can be defined as

$$\begin{array}{c}\hfill \widehat{q}\left(t\right)={\int }_{t-T}^t\left(x^{\mathsf{T}}Qx+M\left(\widehat{u}\left({\widehat{x}}_k\right)\right)-{\gamma }^2{\upsilon }^{\mathsf{T}}\left({\widehat{x}}_k\right)\upsilon \left({\widehat{x}}_k\right)\right)d\tau \end{array}$$

(46)

In the iterative process of NN, its purpose is to minimize the value function. The gradient-descent method is used in this paper. Then, the critic network weights are adjusted online by minimizing the objective function $E=(1/2)e_c^{\mathsf{T}}{e}_c$. Thus, the turning law of the critic NN weight can be expressed as

$$\begin{array}{cc}\hfill {\dot{\widehat{W}}}_c& =-{\alpha }_c\frac{\partial E}{\partial {\widehat{W}}_c}=-{\alpha }_c\frac{\partial E}{\partial e_c}\frac{\partial e_c}{\partial {\widehat{W}}_c}=-{\alpha }_c\frac{\Delta \varphi }{{(1+\Delta {\varphi }^{\mathsf{T}}\Delta \varphi )}^2}\hfill \\ \hfill & \times \left\{{\int }_{t-T}^t\left(x^{\mathsf{T}}Qx+M\left(\widehat{u}\left({\widehat{x}}_k\right)\right)-{\gamma }^2{\upsilon }^{\mathsf{T}}\left({\widehat{x}}_k\right)\upsilon \left({\widehat{x}}_k\right)\right)d\tau +\Delta {\varphi }^{\mathsf{T}}\left(x\right){\widehat{W}}_c\right\}\hfill \end{array}$$

(47)

where ${\alpha }_c>0$ is an adaptive learning rate and ${(1+\Delta {\varphi }^{\mathsf{T}}\Delta \varphi )}^2$ is used for normalization. Define the critic weight approximation error as ${\tilde{W}}_c=W_c-{\widehat{W}}_c$, and the square of the denominator in (47) is used to guarantee ${\tilde{W}}_c$ is bounded. Then, the critic weight error dynamic is presented by

$${\dot{\tilde{W}}}_c=-{\alpha }_c\overline{\Delta }\varphi \left(t\right)\overline{\Delta }{\varphi }^{\mathsf{T}}\left(t\right){\tilde{W}}_c+{\alpha }_c\overline{\Delta }\varphi \left(t\right)m{e}_B\left(t\right)$$

(48)

where $m=1/(1+\Delta {\varphi }^{\mathsf{T}}\Delta \varphi )$, $\overline{\Delta }\varphi =\Delta \varphi /(1+\Delta {\varphi }^{\mathsf{T}}\Delta \varphi )$.

Remark 4.

This paper designs an online event-triggered ADP learning algorithm, the activation function must be continuously excited to ensure that the NN weights can be converged. Then, the value function and control laws are obtained. Therefore, the probing noise signal with different amplitudes and diverse frequencies is constructed as $d\left(t\right)=0.5(sin{\left(0.08t\right)}^{2}cos\left(1.5t\right)+0.3sin{\left(2.3t\right)}^{4}cos\left(7t\right))$.

Remark 5.

Under the triggering condition (23), the adaptive learning law of the critic network (47) updates its weight by the reinforcement signal from (46), then the control law (43), and perturbation law (44) are updated at the triggering instant. In order to intuitively clarify the main idea of the algorithm, Figure 1 shows the block diagram of the event-triggered ADP $H_{\infty }$-constraint control.

Assumption 4

([26]).

• $g\left(x\right)$ and $h\left(x\right)$ are Lipschitz continuous and satisfy $\parallel g\left(x\right)-g\left({\widehat{x}}_k\right)\parallel \le L_g\parallel e_k\left(t\right)\parallel$ and $\parallel h\left(x\right)-h\left({\widehat{x}}_k\right)\parallel \le L_h\parallel e_k\left(t\right)\parallel$.
• $\nabla \varphi \left(x\right)$ is Lipschitz continuous, that is $∥\nabla \varphi \left(x\right)-\nabla \varphi \left({\widehat{x}}_k\right)∥\le L_{\varphi }∥e_k\left(t\right)∥$, $L_{\varphi }$ is a positive constant.

Assumption 5

([34,35,36]).

• The NN activation function and its gradient are bounded by positive constants so that $∥\varphi \left(x\right)∥\le {\varphi }_m$,$∥\nabla \varphi \left(x\right)∥\le {\varphi }_M$.
• The critic NNs approximation error and its gradient are all bounded by a constant and satisfied $∥{\epsilon }_V∥\le {\epsilon }_m$, $∥\nabla {\epsilon }_V∥\le {\epsilon }_M$.

The control law $u\left(x\right)$ is admissible control. The approximate weight ${\widehat{W}}_c$ can be guaranteed to converge to ideal weight $W_c$ under persistent excitation (PE) condition. Assume that the activation function $\Delta \varphi$ is continuously exciting in the interval $[t-T,t]$. Let the residual error due to the approximation error satisfies $∥e_B∥\le e_M$, and it can ensure that the critic network weight is UUB. The boundedness condition of critic NN weight is expressed at (50).

Theorem 3.

Let Assumption 1–5 be valid, and the control laws and the turning law of critic NN are implemented by (43), (44), and (47). The closed-loop system (1) is asymptotically stable and the weight approximation error (48) is UUB only when the triggering condition in (23) is used and the following inequalities condition holds

$$\left({\alpha }_c-\frac{1}{2}\right)\underline{\lambda }\left(\overline{\Delta }\varphi \left(t\right)\overline{\Delta }{\varphi }^{\mathsf{T}}\left(t\right)\right)-{\varpi }_2>0$$

(49)

where ${\varpi }_2$ and ${\varpi }_4$ are constants from (60) and (63).

Proof. 

The initial control for the dynamical system (1) is admissible. For the value function (4) of the system, the Lyapunov function is constructed as

$$\mathcal{L}\left(t\right)={\mathcal{L}}_1+{\mathcal{L}}_2+{\mathcal{L}}_3$$

(50)

where ${\mathcal{L}}_1=V^{*}\left(x\right)$, ${\mathcal{L}}_2=V^{*}\left({\widehat{x}}_k\right)$, ${\mathcal{L}}_3=(1/2){\tilde{W}}_c^{\mathsf{T}}{\tilde{W}}_c$.

The process of NN learning is mainly divided into two cases under event-triggered control: (1) During the flow dynamic on $\mathrm{t}\in [t_k,t_{k+1})$. (2) At the triggering instant $t=t_k$.

Case 1. When the triggering condition is not satisfied and the derivative of ${\mathcal{L}}_1$ can be given as

$$\begin{array}{cc}\hfill {\dot{\mathcal{L}}}_1& =\nabla {V^{*}}^{\mathsf{T}}\left(x\right)\left(f\left(x\right)+g\left(x\right)\widehat{u}\left({\widehat{x}}_k\right)+h\left(x\right)\widehat{\upsilon }\left({\widehat{x}}_k\right)\right)\hfill \\ \hfill & =\nabla {V^{*}}^{\mathsf{T}}\left(x\right)g\left(x\right)\left(\widehat{u}\left({\widehat{x}}_k\right)-u^{*}\left(x\right)\right)+\nabla {V^{*}}^{\mathsf{T}}\left(x\right)h\left(x\right)\left(\widehat{\upsilon }\left({\widehat{x}}_k\right)-{\upsilon }^{*}\left(x\right)\right)\hfill \\ \hfill & +\nabla {V^{*}}^{\mathsf{T}}\left(x\right)\left(f\left(x\right)+g\left(x\right)u^{*}\left(x\right)+h\left(x\right){\upsilon }^{*}\left(x\right)\right)\hfill \end{array}$$

(51)

According to (25)–(27) and (51) can be transformed as

$$\begin{array}{cc}\hfill {\dot{\mathcal{L}}}_1& =2u_{M}R{tanh}^{-\mathsf{T}}\left(\frac{u^{*}\left(x\right)}{u_M}\right)\left(\widehat{u}\left({\widehat{x}}_k\right)-u^{*}\left(x\right)\right)+2{\gamma }^2{{\upsilon }^{*}}^{\mathsf{T}}\left(x\right)\left(\widehat{\upsilon }\left({\widehat{x}}_k\right)-{\upsilon }^{*}\left(x\right)\right)\hfill \\ \hfill & -x^{\mathsf{T}}Qx-M\left(u^{*}\left(x\right)\right)+{\gamma }^2{{\upsilon }^{*}\left(x\right)}^{\mathsf{T}}{\upsilon }^{*}\left(x\right)\hfill \end{array}$$

(52)

Similarly, using Young’s inequality, we have

$$\begin{array}{cc}\hfill {\dot{\mathcal{L}}}_1& \le \parallel u_{M}R{tanh}^{-\mathsf{T}}\left(\frac{u^{*}\left(x\right)}{u_M}\right){\parallel }^2+{∥\widehat{u}\left({\widehat{x}}_k\right)-u^{*}\left(x\right)∥}^2+{\gamma }^2{\parallel {\upsilon }^{*}\left(x\right)\parallel }^2\hfill \\ \hfill & +{\gamma }^2{∥\widehat{\upsilon }\left({\widehat{x}}_k\right)-{\upsilon }^{*}\left(x\right)∥}^2-x^{\mathsf{T}}Qx-M\left(u^{*}\left(x\right)\right)+{\gamma }^2{\upsilon }^{*\mathsf{T}}\left(x\right){\upsilon }^{*}\left(x\right)\hfill \\ \hfill & \le \frac{1}{4}\parallel \nabla V^{*\mathsf{T}}\left(x\right)g{\parallel }^2+2{\gamma }^2{\parallel {\upsilon \left(x\right)}^{*}\parallel }^2-x^{\mathsf{T}}Qx-M\left(u^{*}\left(x\right)\right)\hfill \\ \hfill & +\underset{o_1}{\underbrace{{∥\widehat{u}\left({\widehat{x}}_k\right)-u^{*}\left(x\right)∥}^2}}+{\gamma }^2\underset{o_2}{\underbrace{{∥\widehat{\upsilon }\left({\widehat{x}}_k\right)-{\upsilon }^{*}\left(x\right)∥}^2}}\hfill \end{array}$$

(53)

Combining (8) with (38), then we obtain

$$\begin{array}{cc}\hfill u^{*}\left(x\right)& =-u_{M}tanh\left(\frac{1}{2u_M}{R}^{-1}{g}^{\mathsf{T}}\left(x\right)\nabla V^{*}\left(x\right)\right)\hfill \\ \hfill & =-u_{M}tanh\left(\frac{1}{2u_M}{R}^{-1}{g}^{\mathsf{T}}\left(x\right)\nabla {\varphi }^{\mathsf{T}}\left(x\right)\left\{W_c+\nabla {\epsilon }_V\left(x\right)\right\}\right)\hfill \end{array}$$

(54)

Note that $|tanh(x\left)\right|\le 1$ and $|x-y|\le \left|x\right|+\left|y\right|$, Then, we have

$$\begin{array}{cc}\hfill o_1& =\parallel u_{M}tanh\left(\frac{1}{2u_M}{R}^{-1}{g}^{\mathsf{T}}\left(x\right)\left(\nabla {\varphi }^{\mathsf{T}}\left(x\right)W_c+\nabla {\epsilon }_V\left(x\right)\right)\right)\hfill \\ \hfill & -u_{M}tanh\left(\frac{1}{2u_M}{R}^{-1}{g}^{\mathsf{T}}\left({\widehat{x}}_k\right)\nabla {\varphi }^{\mathsf{T}}\left({\widehat{x}}_k\right)(W_c-{\tilde{W}}_c)\right){\parallel }^2\hfill \\ \hfill & \le u_M^2\parallel \frac{1}{2u_M}{R}^{-1}{g}^{\mathsf{T}}\left(x\right)\left(\nabla {\varphi }^{\mathsf{T}}\left(x\right)W_c+\nabla {\epsilon }_V\left(x\right)\right)+\frac{1}{2u_M}{R}^{-1}{g}^{\mathsf{T}}\left({\widehat{x}}_k\right)\nabla {\varphi }^{\mathsf{T}}\left({\widehat{x}}_k\right)({\tilde{W}}_c-W_c){\parallel }^2\hfill \\ \hfill & \le \frac{1}{4}{\parallel R^{-1}\parallel }^2\parallel \left(\nabla \varphi \left(x\right)g\left(x\right)-\nabla \varphi \left({\widehat{x}}_k\right)g\left({\widehat{x}}_k\right)\right)W_c+g^{\mathsf{T}}\left(x\right)\nabla {\epsilon }_V\left(x\right)+\nabla \varphi \left({\widehat{x}}_k\right)g\left({\widehat{x}}_k\right){\tilde{W}}_c{\parallel }^2\hfill \\ \hfill & \le \frac{1}{2}{\parallel R^{-1}\parallel }^2\left(\underset{{\mathcal{F}}_1}{\underbrace{{∥\nabla \varphi \left(x\right)g\left(x\right)-\nabla \varphi \left({\widehat{x}}_k\right)g\left({\widehat{x}}_k\right)∥}^2}}{∥W_c∥}^2+\underset{{\mathcal{F}}_2}{\underbrace{{∥g^{\mathsf{T}}\left(x\right)\nabla {\epsilon }_V\left(x\right)+\nabla \varphi \left({\widehat{x}}_k\right)g\left({\widehat{x}}_k\right){\tilde{W}}_c∥}^2}}\right)\hfill \end{array}$$

(55)

According to Assumption 4 and Assumption 5, we obtain

$$\begin{array}{cc}\hfill {\mathcal{F}}_1& ={∥\nabla \varphi \left(x\right)g\left(x\right)-\nabla \varphi \left({\widehat{x}}_k\right)g\left(x\right)+\nabla \varphi \left({\widehat{x}}_k\right)g\left(x\right)-\nabla \varphi \left({\widehat{x}}_k\right)g\left({\widehat{x}}_k\right)∥}^2\hfill \\ \hfill & ={∥\left(\nabla \varphi \left(x\right)-\nabla \varphi \left({\widehat{x}}_k\right)\right)g\left(x\right)+\nabla \varphi \left({\widehat{x}}_k\right)\left(g\left(x\right)-g\left({\widehat{x}}_k\right)\right)∥}^2\hfill \\ \hfill & \le 2{∥\left(\nabla \varphi \left(x\right)-\nabla \varphi \left({\widehat{x}}_k\right)\right)g\left(x\right)∥}^2+2{∥\nabla \varphi \left({\widehat{x}}_k\right)\left(g\left(x\right)-g\left({\widehat{x}}_k\right)\right)∥}^2\hfill \\ \hfill & \le 2\left(L_{\varphi }^2{g}_M^2+{\varphi }_M^2{L}_g^2\right){∥e_k∥}^2\hfill \end{array}$$

(56)

Similarly, it can be obtained

$$\begin{array}{cc}\hfill {\mathcal{F}}_2& ={∥g^{\mathsf{T}}\left(x\right)\nabla {\epsilon }_V\left(x\right)+\nabla \varphi \left({\widehat{x}}_k\right)g\left({\widehat{x}}_k\right){\tilde{W}}_c∥}^2\hfill \\ \hfill & \le 2{∥g^{\mathsf{T}}\left(x\right)\nabla {\epsilon }_V\left(x\right)∥}^2+2{∥\nabla \varphi \left({\widehat{x}}_k\right)g\left({\widehat{x}}_k\right){\tilde{W}}_c∥}^2\hfill \\ \hfill & \le 2\left(g_M^2{\epsilon }_M^2+g_M^2{\varphi }_M^2{∥{\tilde{W}}_c∥}^2\right)\hfill \end{array}$$

(57)

According to (55)–(57), $o_1$ rewrite as

$$o_1\le \parallel R^{-1}{\parallel }^2\left(L_{\varphi }^2{g}_M^2+{\varphi }_M^2{L}_g^2\right){∥e_k∥}^2\parallel W_c{\parallel }^2+{\parallel R^{-1}\parallel }^2(g_M^2{\epsilon }_M^2+g_M^2{\varphi }_M^2{∥{\tilde{W}}_c∥}^2)$$

(58)

Lisewise, combining (9) with (37), $o_2$ can be given as

$$\begin{array}{cc}\hfill o_2& \le \parallel \frac{1}{2}{\gamma }^{-2}{h}^{\mathsf{T}}\left(x\right)\left(\nabla {\varphi }^{\mathsf{T}}\left(x\right)W_c+\nabla {\epsilon }_V\left(x\right)\right)-\frac{1}{2}{\gamma }^{-2}{h}^{\mathsf{T}}\left({\widehat{x}}_k\right)\nabla {\varphi }^{\mathsf{T}}\left({\widehat{x}}_k\right){\widehat{W}}_c{\parallel }^2\hfill \\ \hfill & \le \frac{1}{2}{∥\nabla \varphi \left(x\right)h\left(x\right)-\nabla \varphi \left({\widehat{x}}_k\right)h\left({\widehat{x}}_k\right)∥}^2\parallel W_c{\parallel }^2+\parallel h^{\mathsf{T}}\left(x\right)\nabla {\epsilon }_V{\left(x\right)\parallel }^2+{\parallel \nabla \varphi \left({\widehat{x}}_k\right)h\left({\widehat{x}}_k\right){\tilde{W}}_c\parallel }^2\hfill \\ \hfill & \le (L_{\varphi }^2{h}_M^2+{\varphi }_M^2{L}_h^2){∥e_k∥}^2{\parallel W_c\parallel }^2+h_M^2{\epsilon }_M^2+h_M^2{\varphi }_M^2{∥{\tilde{W}}_c∥}^2\hfill \end{array}$$

(59)

According to (53), (58) and (59), we have

$$\begin{array}{cc}\hfill {\dot{\mathcal{L}}}_1& \le -x^{\mathsf{T}}Qx-M\left(u^{*}\left(x\right)\right)+{\varpi }_1{∥e_k∥}^2{\parallel W_c\parallel }^2+{\varpi }_2{\parallel {\tilde{W}}_c\parallel }^2+{\varpi }_3\hfill \\ \hfill & \le -(1-{\sigma }^2)\underline{\lambda }\left(Q\right){∥x∥}^2-{\sigma }^2\underline{\lambda }\left(Q\right){∥x∥}^2+{\varpi }_1{∥e_k∥}^2{\parallel W_c\parallel }^2+{\varpi }_2{\parallel {\tilde{W}}_c\parallel }^2+{\varpi }_3\hfill \end{array}$$

(60)

where ${\varpi }_1=\parallel R^{-1}{\parallel }^2\left(L_{\varphi }^2{g}_M^2+{\varphi }_M^2{L}_g^2\right)+{\gamma }^2(L_{\varphi }^2{h}_M^2+{\varphi }_M^2{L}_h^2),{\varpi }_2=\parallel R^{-1}{\parallel }^2{g}_M^2{\varphi }_M^2+{\gamma }^2{h}_M^2{\varphi }_M^2,{\varpi }_3=\frac{1}{4}{g}_M^2{\eta }_0+2{\gamma }^2{\upsilon }_M^2+{\parallel R^{-1}\parallel }^2{g}_M^2{\epsilon }_M^2+{\gamma }^2{h}_M^2{\epsilon }_M^2.$

If the event is not triggered, we have ${\dot{\mathcal{L}}}_2=0$. Note that $1/(1+\Delta {\varphi }^{\mathsf{T}}\Delta \varphi )\le 1$, thus, m, $\overline{\Delta }\varphi \left(t\right)$ from (48) are bounded. Then, according to $W_c={\widehat{W}}_c+{\tilde{W}}_c$, the part of ${\dot{\mathcal{L}}}_3$ can be given as

$$\begin{array}{cc}\hfill {\dot{\mathcal{L}}}_3& ={\tilde{W}}_c^{\mathsf{T}}{\dot{\tilde{W}}}_c=-{\alpha }_c{{\tilde{W}}_c}^{\mathsf{T}}\overline{\Delta }\varphi \left(t\right)\overline{\Delta }{\varphi }^{\mathsf{T}}\left(t\right){\tilde{W}}_c+{\alpha }_c{\tilde{W}}_c^{\mathsf{T}}\overline{\Delta }\varphi \left(t\right)m{\epsilon }_B\left(t\right)\hfill \\ \hfill & \le -{\alpha }_c\underline{\lambda }\left(\overline{\Delta }\varphi \left(t\right)\overline{\Delta }{\varphi }^{\mathsf{T}}\left(t\right)\right)\parallel {\tilde{W}}_c{\parallel }^2+\frac{1}{2}{\alpha }_c^2{e}_M^2+\frac{1}{2}\underline{\lambda }\left(\overline{\Delta }\varphi \left(t\right)\overline{\Delta }{\varphi }^{\mathsf{T}}\left(t\right)\right){\parallel {\tilde{W}}_c\parallel }^2\hfill \\ \hfill & \le -({\alpha }_c-\frac{1}{2})\underline{\lambda }\left(\overline{\Delta }\varphi \left(t\right)\overline{\Delta }{\varphi }^{\mathsf{T}}\left(t\right)\right){\parallel {\tilde{W}}_c\parallel }^2+\frac{1}{2}{\alpha }_c^2{e}_M^2\hfill \end{array}$$

(61)

Combining (60) and (61), we have

$$\begin{array}{cc}\hfill \dot{\mathcal{L}}\le & -(1-{\sigma }^2)\underline{\lambda }\left(Q\right){∥x∥}^2-{\sigma }^2\underline{\lambda }\left(Q\right){∥x∥}^2+{\varpi }_1{∥e_k∥}^2{\parallel W_c\parallel }^2+{\varpi }_2{\parallel {\tilde{W}}_c\parallel }^2\hfill \\ \hfill & -\left({\alpha }_c-\frac{1}{2}\right)\underline{\lambda }\left(\overline{\Delta }\varphi \left(t\right)\overline{\Delta }{\varphi }^{\mathsf{T}}\left(t\right)\right){\parallel {\tilde{W}}_c\parallel }^2+\frac{1}{2}{\alpha }_c^2{e}_M^2+{\varpi }_3\hfill \end{array}$$

(62)

Let the triggering condition (23) holding the derivative of $\mathcal{L}$ become

$$\begin{array}{c}\hfill \dot{\mathcal{L}}\le -{\sigma }^2\underline{\lambda }\left(Q\right){∥x∥}^2-\left(\left({\alpha }_c-\frac{1}{2}\right)\underline{\lambda }\left(\overline{\Delta }\varphi \left(t\right)\overline{\Delta }{\varphi }^{\mathsf{T}}\left(t\right)\right)-{\varpi }_2\right){\parallel {\tilde{W}}_c\parallel }^2+{\varpi }_4\end{array}$$

(63)

where ${\varpi }_4=(1/2){\alpha }_c^2{e}_M^2+{\varpi }_3$. Let the triggering condition (23) and the inequality condition (49) hold. We can conclude that $\dot{\mathcal{L}}<0$ if the $∥x\left(t\right)∥$ and $\parallel {\tilde{W}}_c\parallel$ are satisfied with

$$∥x∥>\sqrt{\frac{{\varpi }_4}{{\sigma }^2\underline{\lambda }\left(Q\right)}}$$

(64)

$$∥{\tilde{W}}_c∥>\sqrt{\frac{2{\varpi }_4}{\left(2{\alpha }_c-1\right)\underline{\lambda }\left(\overline{\Delta }\varphi \left(t\right)\overline{\Delta }{\varphi }^{\mathsf{T}}\left(t\right)\right)-2{\varpi }_2}}$$

(65)

Therefore, by the Lyapunov theory, the state x and the NN weight error ${\tilde{W}}_c$ are uniformly ultimately bounded.

Case 2. The event is triggered at $t=t_{k+1}$. A new Lyapunov function is constructed as

$$\Delta \mathcal{L}\left(t\right)=\Delta {\mathcal{L}}_1+\Delta {\mathcal{L}}_2+\Delta {\mathcal{L}}_3$$

(66)

where $\Delta {\mathcal{L}}_1=V^{*}\left(x^{+}\right)-V^{*}\left({\widehat{x}}_k\right)$, $\Delta {\mathcal{L}}_2=V^{*}\left({\widehat{x}}_{k+1}\right)-V^{*}\left({\widehat{x}}_k\right)$, $\Delta {\mathcal{L}}_3=1/2(\tilde{W}{{}_c^{+}}^{\mathsf{T}}{\tilde{W}}_c^{+}-{\tilde{W}}_c^{\mathsf{T}}{\tilde{W}}_c)$. If $\xi \to 0^{+}$, then we have $x^{+}=x\left(t_k^{+}\right)\to x(t_k+\xi )$ with $\xi \in (0,t_{k+1}-t_k)$. It can be concluded x and ${\tilde{W}}_c$ are UUB in the flow dynamic from Case 1 such that the closed-loop system (1) is asymptotically stable. Since $V^{*}\left(x\right)$ and ${\tilde{W}}_c$ are continuous, then we can obtain $\Delta {\mathcal{L}}_1=0$ and $\Delta {\mathcal{L}}_3=0$. At the triggering instant $t=t_{k+1}$, we have

$$\Delta {\mathcal{L}}_2=V^{*}\left({\widehat{x}}_{k+1}\right)-V^{*}\left({\widehat{x}}_k\right)\le -\omega (\parallel e_{k+1}\left(t_k\right)\parallel )<0$$

(67)

where $e_{k+1}\left(t_k\right)={\widehat{x}}_{k+1}-{\widehat{x}}_k$, and $\omega (·)$ is a class-$\kappa$ function [43]. Therefore, we can obtain $\Delta \mathcal{L}<0$. The above two cases ensure that the closed-loop system under the event-triggered strategy is asymptotically stable and the NN weight error is UUB if the triggering condition (23) and the inequality (49) are held. The proof is completed. □

Remark 6.

The theoretical description has been shown. Algorithm 2 can be implemented by using a single critic NN, which updates the critic weight using the turning law (47). Then, the new control laws $u\left(x\right)$ and $\upsilon \left(x\right)$ are obtained by (43) and (44) when an event occurs. Compared with [12,26,35,44], we propose the event-triggered ADP method which solves the zero-sum problem with actuator saturation and partially unknown dynamic. Based on event-triggered control, the control laws of the two-player are presented in segments by the designed triggering condition (23). Thus, the event-triggered control avoids unnecessary data transmission and calculation costs.

5. Simulation Results

In this section, two simulation examples of the linear system and the nonlinear system are provided to verify the effectiveness of the online event-triggered adaptive dynamic programming algorithm for the continuous-time system.

5.1. Linear Sytsem

Consider the F-16 aircraft continuous-time linear system with external perturbation as [12]

$$\dot{x}=Ax\left(t\right)+Bu\left(t\right)+H\upsilon \left(t\right)$$

(68)

where

$A=\left[\begin{array}{ccc}-1.01887& 0.90506& -0.00215\\ 0.82225& -1.07741& -0.17555\\ 0& 0& -1\end{array}\right]$, $B={[0,0,1]}^{\mathsf{T}}$, $H={\left[1,0,1\right]}^{\mathsf{T}}$

The three state vectors are denoted as $x={[x_1,x_2,x_3]}^{\mathsf{T}}$, and the initial state is $x_0={[1,-1,2]}^{\mathsf{T}}$. Choose a matrix of the appropriate dimension, where $Q=I_3$ and $R=I_1$. The constrained control can be satisfied with $u\in \{u\in \mathbb{R}:∥u∥\le 1\}$. The activation function of single NNs is $\varphi \left(x\right)={[x_1^2,x_2^2,x_3^2,x_1{x}_2,x_1{x}_3,x_2{x}_3]}^{\mathsf{T}}$. Then, the approximated weight vectors are denoted as ${\widehat{W}}_c=[{\widehat{W}}_{c_1},{\widehat{W}}_{c_2},{\widehat{W}}_{c_3},{\widehat{W}}_{c_4},{\widehat{W}}_{c_5},{\widehat{W}}_{c_6}]$, which is the solution of the algebraic Riccati equation (ARE): $A^{\mathsf{T}}P+PA-PB{R}^{-1}{B}^{\mathsf{T}}P+Q=0$. The NN learning and triggering parameters are needed in this paper: $\gamma =0.5$, $L\in (0,1)$, $T=0.1$, ${\alpha }_c=0.35$. The external disturbance is selected as $\upsilon \left(t\right)=5cos\left(t\right)e^{-t}$.

Figure 2 shows the convergence of three state vectors. It is obvious that the states tend to be asymptotically stable. The iterative process of critic NN is illustrated in Figure 3. The critic weight vectors are converged to ${\widehat{W}}_c=[-$0.0349,−0.0128,−0.1575, 1.4738, 0.6577, 0.2075${]}^{\mathsf{T}}$ under the PE condition after 60s. Therefore, the PE condition is effectively guaranteed by adding detection noise. Under the event-triggered strategy, the control law and the perturbation law are updated in a nonperiodical manner. The evolution of ETC laws $u\left({\widehat{x}}_k\right)$ and $\upsilon \left({\widehat{x}}_k\right)$ is described in Figure 4. Under the condition of constrained input, the control law remains no more than 1, that is $\parallel u_{max}\parallel <1$. The trajectory of event error (15) and the triggering threshold (23) is shown in Figure 5, which verifies the working effect of the triggering condition. Figure 6 presents the event-triggered sample intervals and periods under the triggering condition. Compared with 1000 sampling states based on the time-triggered strategy, only 85 state samples are performed based on the event-triggered strategy. From

Table 1. State Sampling of Two Strategies.

Methods	Event-Triggered	Time-Triggered
Samples	85	1000
Minimal interval (s)	0.1	0.1
Average interval (s)	1.0165	0.1

, the minimum interval time ${\left({\tau }_k\right)}_{min}=0.1$, and the Zeno phenomenon is avoided. As a result, the updated frequencies and computations of the controller are greatly reduced.

5.2. Nonlinear Sytsem

Consider a continuous-time nonlinear system with the external disturbance given as

$$\dot{x}=f\left(x\left(t\right)\right)+g\left(x\left(t\right)\right)u\left(t\right)+h\left(x\left(t\right)\right)\upsilon \left(t\right)$$

(69)

where

$f\left(x\right)=\left[\begin{array}{c}-x_1+x_2\\ -0.5x_1-0.5x_2\left(1-cos{(\left(2x_1\right)+2)}^2\right)\end{array}\right]$

$g\left(x\right)=\left[\begin{array}{c}0\\ cos\left(2x_1\right)+2\end{array}\right]$, $h\left(x\right)=\left[\begin{array}{c}0\\ sin(4x_1+2)\end{array}\right]$

The state vectors are denoted as $x={[x_1,x_2]}^{\mathsf{T}}$, and the initial state is $x_0={[1,-1]}^{\mathsf{T}}$. The constrained input control is satisfied with $u\in \{u\in \mathbb{R}:∥u∥\le 1\}$. Choose $Q=I_2$ and $R=I_1$. The activation function is selected as $\varphi \left(x\right)={[x_1^2,x_1{x}_2,x_2^2]}^{\mathsf{T}}$. Then, the approximated weight vectors are denoted as ${\widehat{W}}_c=[{\widehat{W}}_{c_1},{\widehat{W}}_{c_2},{\widehat{W}}_{c_3}]$. The NN learning and triggering parameters are needed in this paper: $\gamma =0.4$, $L\in (0,1)$, $T=0.1$, ${\alpha }_c=0.25$. The external disturbance is selected as $\upsilon \left(t\right)=5cos\left(t\right)e^{-t}$ similarly.

The two state trajectories of the closed-loop system are described in Figure 7. It is obvious that $x_1$ and $x_2$ gradually converge to asymptotically stable. The ideal critic weight is ${[W_{c_1},W_{c_2},W_{c_3}]}^{\mathsf{T}}$. The NNs can be ensured to converge under the action of PE condition. Figure 8 demonstrates the approximable critic weight vectors are converged to ${[0.5468,0.8876,-0.4125]}^{\mathsf{T}}$ after learning. From Figure 9, we can obtain the evolution of ETC policies $u\left({\widehat{x}}_k\right)$ and $\upsilon \left({\widehat{x}}_k\right)$. Under the condition of constrained input, the control law remains no more than 1, that is $\parallel u_{max}\parallel <1$. The trajectory of the event error and the triggering threshold are shown in Figure 10. It can be seen intuitively that the event-triggered errors are always within the triggering thresholds. Figure 11 indicates the event-triggered sampling intervals under the event-triggered mechanism. From

Table 2. Comparison of Nonlinear System Sampling.

Methods	Event-Triggered	Time-Triggered
Samples	157	1000
Minimal interval (s)	0.1	0.1
Average interval (s)	0.527	0.1

, it has only 157 sampling states are performed based on an event-triggered strategy. Likewise, states will be sampled at each time by using a time-triggered strategy.

6. Conclusions

This paper has proposed an event-triggered ADP algorithm to solve the locally unknown zero-sum game problem with constrained input. Using the IRL technique to obtain the unknown dynamic system, and a single-critic NN is structured to solve the HJI equation. The event-based $H_{\infty }$ controller is designed to ensure that the state of the system is sampled only at the triggering instant. Furthermore, the triggering condition guarantees the dynamic system state and the critic weight are uniformly ultimately bounded. Two examples are used to verify the feasibility of the algorithm. However, with the complexity and nonlinearity of the system, this algorithm is not enough to solve systems with model-free or parameter uncertainties. Therefore, the direction of future research is how to solve the above problem.