Improved Q-Learning Method for Linear Discrete-Time Systems

Abstract

In this paper, the Q-learning method for quadratic optimal control problem of discrete-time linear systems is reconsidered. The theoretical results prove that the quadratic optimal controller cannot be solved directly due to the linear correlation of the data sets. The following corollaries have been made: (1) The correlation of data is the key factor in the success for the calculation of quadratic optimal control laws by Q-learning method; (2) The control laws for linear systems cannot be derived directly by the existing Q-learning method; (3) For nonlinear systems, there are some doubts about the data independence of current method. Therefore, it is necessary to discuss the probability of the controllers established by the existing Q-learning method. To solve this problem, based on the ridge regression, an improved model-free Q-learning quadratic optimal control method for discrete-time linear systems is proposed in this paper. Therefore, the computation process can be implemented correctly, and the effective controller can be solved. The simulation results show that the proposed method can not only overcome the problem caused by the data correlation, but also derive proper control laws for discrete-time linear systems.

1. Introduction

This article reconsiders the Q-learning method for quadratic optimal control problem of discrete-time linear systems. Therefore, it is necessary to explain the quadratic optimal control problem of discrete-time linear systems. There is a discrete-time linear system which can be described as $\mathbf{x}(l+1)=\mathbf{Ax}\left(l\right)+\mathbf{Bu}\left(l\right)$, where $\mathbf{x}\left(l\right)$ is states of system, $\mathbf{u}\left(l\right)$ is the control value, $\mathbf{A}$ is state matrix, $\mathbf{B}$ is control matrix. The quadratic optimal control problem for the system is to minimize the quadratic performance index function $\mathit{J}={\sum }_{i=l}^{\infty }({\mathbf{x}}^T\left(i\right)\mathbf{Q}\mathbf{x}\left(i\right)+{\mathbf{u}}^T\left(i\right)\mathbf{R}\mathbf{u}\left(i\right))$, by designing a proper state feedback matrix $\mathbf{K}$ and choosing control law $\mathbf{u}\left(l\right)=-\mathbf{Kx}\left(l\right)$. According to the optimal control theory, the optimal value of feedback matrix $\mathbf{K}$ should satisfy following equation: $\mathbf{K}={(\mathbf{R}+{\mathbf{B}}^T\mathbf{PB})}^{-1}{\mathbf{B}}^T\mathbf{PA}$, where the matrix $\mathbf{P}$ should be the solution of Riccati equation $\mathbf{P}={\mathbf{A}}^T\mathbf{PA}+\mathbf{Q}-{\mathbf{A}}^T\mathbf{PB}{(\mathbf{R}+{\mathbf{B}}^T\mathbf{PB})}^{-1}{\mathbf{B}}^T\mathbf{PA}$. So, if matrices $\mathbf{A}$ and $\mathbf{B}$ are known, by setting proper matrices $\mathbf{Q}$ and $\mathbf{R}$, and resolving matrix from Riccati equation, the optimal feedback matrix $\mathbf{K}$ can be solved. It can be seen from the above process that the precondition of the calculation process is the precise acquaintance of controlled systems’ mathematical models.

There are great amount of papers which discussed the design of model-free controllers via Q-learning method for linear discrete-time systems, while the independence of data during the computation process is not get enough attention. In this paper, the effect of the data correlation is discussed, specifically aiming to the computation processes of optimal controllers for linear discrete-time systems adopting Q-learning method. Based on the theoretical analysis, the data independence condition should be satisfied to implement the existing Q-learning method. The data sets sampled from the linear discrete-time systems is destined relevant, which means that the controllers cannot be solved by the existing Q-learning method. To design proper controllers for linear discrete-time systems, an improved Q-learning method is proposed in this paper. The improved method adopts the ridge regression instead of the least squares regression. The ridge regression can deal with the multiple collinearity of the sample sets. Therefore, the proper controllers can be solved by the proposed method.

In this paper, following results have been achieved. First, limited to linear discrete-time systems, the computation process cannot be executed correctly by the existing Q-learning method. Second, relative corollaries have been made for several types of systems, to indicate in detail the necessary for data independence condition for different situations. Third, an improved Q-learning method has been proposed to solve the problem of multiple collinearity. At last, results of simulations show the effectiveness of the proposed method.

2. Q-Learning Method for Model-Free Control Schemes

Generally, there is an obvious difference between optimal control [1] and adaptive control [2,3]. Adaptive controller often needs online dynamic data of systems. On the contrary, optimal controllers are designed off-line by solving Hamilton-Jacobi-Bellman (HJB) equations. In optimal theory, the solution of HJB equations is the real valued function with minimum cost for specific dynamic system and corresponding cost function. Therefore, the solution of HJB equations is also the optimal controller of the controlled system. In the HJB equations, all information of the controlled system should be known, which means the precise mathematical model of the system should be established. To solve HJB equations, many methods have been researched, and the earlier mentioned Riccati equation is derived from HJB equations. According former expoundation, the optimal controller cannot be designed under model-free condition. However, the development of theory made the combination of the two different control schemes. The reinforcement learning theory plays an important part in the process.

As one of the most important reinforcement learning methods, Q-learning has been proved by Watkins [4] that under certain conditions the algorithm converges to the optimum cation-values. Therefore, the calculation of optimal controllers became realizable for complete or partial model-free systems by adopting online calculation method. With data sets sampled from systems, optimal controllers can be designed by Q-learning method. Many researchers took the view and many achievements have been made. Compared with other popular techniques such as SVM and KNN [5], reinforcement learning methods represented by Q-learning are unsupervised algorithms. They do not need the target information of the train data, on the contrary, they evaluate their actions by the value functions aiming to the current status. Their directions of calculation are decided by the value functions, in this sense, the reinforcement learning methods are also gradient-free optimization methods [6] when they are used to solve optimization problems. Some optimal estimation algorithms such as Kalman filter and its extended forms [7] have been adopted by many researchers as observers in adaptive control schemes. Their observation results still rely on the structure and parameters of systems, so these algorithms are beyond the boundary of model-free methods.

Bradtke used the Q-learning method to solve quadratic control problem for discrete-time linear systems [8]. He set up the model-free quadratic optimal control solution based on data, and studied the convergence of Q-learning method for certain type of systems. From then on, the important position of Q-learning in model-free quadratic optimal control is established. And the Q-learning schemes for the continuous-time systems also had been proposed [9,10], without the convergence condition.

As a kind of typical data-based iterative algorithm, Q-learning method has been used to solve problems in many fields, including but not limited to following examples. An adaptive Q-learning algorithm is adopted to obtain the optimal solution between exploration and exploitation of electricity market [11]. An effective Q-learning algorithm is proposed to solve the traffic signal control problem [12], and satisfying results have been obtained. A distributed Q-learning (QD-learning) [13] algorithm is investigated to get the solution of the sensor networks collaboration problem. A time-based Q-learning algorithm is adopted to get the optimal control of the energy systems [14,15].

Compared with other fields, the quadratic optimal control problems attract main attention in this paper. To design feedback controllers for discrete and continuous-time systems, the principles of using reinforcement learning method are fully described [16]. According to the paper, reinforcement learning method of policy iteration and value iteration can be used to design adaptive controllers which converge to the optimal control theory. Adopting Q-learning schemes for adaptive linear quadratic controllers [17], the difference between linear and nonlinear discrete-time systems has been compared. Simulations show that controllers for both kind of the systems can be solved with almost same convergence speed. Some researchers focused on output feedback control schemes [18,19,20,21]. The method to design optimal or nearly optimal controllers has been studied for linear discrete-time systems subject to actuator saturation condition [21,22,23,24]. Other researchers considered stochastic linear quadratic optimal control as an effective method [25,26,27,28].

3. Problem Description and Improved Method

Theoretically, the existing Q-learning algorithm can calculate optimal or approximate optimal controllers as a model-free method. For discrete-time linear systems, there is an obvious limitation caused by multi-collinearity of data sets sampled from systems, which is shown by analysis. To solve the problem, the existing Q-learning algorithm need to be modified, as the same situation as the existing stability analysis method for uncertain retarded systems [29]. By adopting ridge regression, an improved algorithm has been proposed in this paper, which can overcome the multi-collinearity problem and obtain the proper controller. To explain the limitation of the existing Q-learning method, the design principle of the existing Q-learning model-free quadratic optimal controllers is shown in following section.

3.1. Design Process of Quadratic Optimal Controller by Existing Q-Learning Method

To design a quadratic optimal controller, a linear discrete-time system is considered to be:

$$x(l+1)=\mathbf{A}x\left(l\right)+\mathbf{B}u\left(l\right)$$

(1)

where $x\in R^n$ is a vector of state variables, $u\in R^p$ is control input vector, $\mathbf{A}$ is $n\times n$ state matrix, $\mathbf{B}$ is input $n\times p$ matrix. And $(\mathbf{A},\mathbf{B})$ is a complete controllability pair. There is the single step performance index is expressed as:

$$\nu \left(l\right)=x^T\left(l\right)\mathbf{Q}x\left(l\right)+u^T\left(l\right)\mathbf{R}u\left(l\right)$$

(2)

where $\mathbf{Q}={\mathbf{Q}}^T\ge 0$ is $n\times n$ weight matrix, $\mathbf{R}>0$ is $p\times p$ weight matrix, $({\mathbf{Q}}^{\frac{1}{2}},\mathbf{A})$ is complete observability pair, $0\le \gamma \le 1$ is discount factor. Please note that:

$$\mathbf{V}\left(x\left(l\right)\right)=\sum _{i=1}^{\infty }{\gamma }^{i-1}\nu \left(i\right)$$

(3)

The problem of quadratic optimal control can be described as follows. For system formed as Equation (1), a control law $u=\mathbf{K}x$ should be designed to minimize $\mathbf{V}\left(x\left(0\right)\right)={\sum }_{i=1}^{\infty }{\gamma }^{i-1}\nu \left(i\right)$. Based on Formula (3), $\mathbf{V}\left(x\right(0\left)\right)$ means the total performance index for infinite time quadratic control systems. To minimize the total performance index, the Q function is defined as:

$$Q\left(x\right(l),u(l\left)\right)=\nu \left(l\right)+\gamma \mathbf{V}\left(x\right(l+1\left)\right)$$

(4)

Formula (4) shows that the Q function is not a new function but just another expression of total performance index $\mathbf{V}\left(x\right(l\left)\right)$. Then the Q function can be calculated as following formula:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill Q\left(x\right(l),u(l\left)\right)=& {\left[\begin{array}{c}x\left(l\right)u\left(l\right)\end{array}\right]}^T\mathbf{H}\left[\begin{array}{c}x\left(l\right)u\left(l\right)\end{array}\right]={\left[\begin{array}{c}x\left(l\right)u\left(l\right)\end{array}\right]}^T\left[\begin{array}{ccc}{\mathbf{H}}_{xx}& {\mathbf{H}}_{xu}{\mathbf{H}}_{ux}& {\mathbf{H}}_{uu}\end{array}\right]\left[\begin{array}{c}x\left(l\right)u\left(l\right)\end{array}\right]\hfill \\ \hfill & ={\left[\begin{array}{c}x\left(l\right)u\left(l\right)\end{array}\right]}^T\left[\begin{array}{ccc}\mathbf{Q}+\gamma {\mathbf{A}}^T\mathbf{PA}& \gamma {\mathbf{A}}^T\mathbf{PB}\gamma {\mathbf{B}}^T\mathbf{PA}& \mathbf{R}+\gamma {\mathbf{B}}^T\mathbf{PB}\end{array}\right]\left[\begin{array}{c}x\left(l\right)u\left(l\right)\end{array}\right]\hfill \end{array}\end{array}$$

(5)

The feedback law can be calculated by following formula according to the extremum condition of $Q\left(x\right(l),u(l\left)\right)$,which is $\partial Q/\partial u=0$.

$$\mathbf{K}=-{\mathbf{H}}_{uu}^{-1}{\mathbf{H}}_{ux}=-\gamma {(\mathbf{R}+{\mathbf{B}}^T\mathbf{PB})}^{-1}{\mathbf{B}}^T\mathbf{PA}$$

(6)

where $\mathbf{P}$ is the solution of Riccati equation:

$$\mathbf{P}=\gamma {\mathbf{A}}^T\mathbf{PA}+\mathbf{Q}-\gamma {\mathbf{A}}^T\mathbf{PB}(\mathbf{R}+\gamma {\mathbf{B}}^T\mathbf{PB}){\mathbf{B}}^T\mathbf{PA}$$

(7)

And $V\left(x\right(l\left)\right)$ should satisfy following equation: $V\left(x\left(l\right)\right)=x^T\left(l\right)\mathbf{P}x\left(l\right)$.

The design method of model-based controller is that the control law will be established by solving Equation (7). According to the optimal control theory, the values of the matrices and will decide the different weights between the quantity of states and control values [26]. The values of matrices of Q and R often depend on performance index, experience of designers and constraint conditions. The quadratic optimal controller based on Q-learning is model-free, and shown as follows. Equation (5) is represented as:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill Q\left(x\right(l),u(l\left)\right)=& {\left[\begin{array}{c}x\left(l\right)u\left(l\right)\end{array}\right]}^T\mathbf{H}\left[\begin{array}{c}x\left(l\right)u\left(l\right)\end{array}\right]={\overline{x}}^T\mathbf{H}\overline{x}\hfill \\ \hfill & =\sum _{i=1}^{n+p}{h}_{ij}{\overline{x}}_i^2+\sum _{i=1}^{n+p}\sum _{j=i+1}^{n+p}2h_{ij}{\overline{x}}_i{\overline{x}}_j={\phi }^T\left(l\right)\Theta \left(\mathbf{H}\right)\hfill \end{array}\end{array}$$

(8)

where ${\phi }^T=\left[\begin{array}{cccccccccc}{\overline{x}}_1^2& {\overline{x}}_1{\overline{x}}_2& \cdots & {\overline{x}}_1{\overline{x}}_{n+p}& {\overline{x}}_2^2& {\overline{x}}_2{\overline{x}}_3& \cdots & {\overline{x}}_2{\overline{x}}_{n+p}& \cdots & {\overline{x}}_{n+p}^2\end{array}\right]$, and the parameters vector $\Theta \left(\mathbf{H}\right)=\left[\begin{array}{cccccccccc}{h}_{11}& 2h_{12}& \cdots & 2h_{1(n+p)}& h_{22}& 2h_{23}& \cdots & 2h_{2(n+p)}& \cdots & h_{(n+p)(n+p)}\end{array}\right]$. The Q function possesses the recurrence relation shown by following formula:

$$Q\left(x\right(l),u(l\left)\right)=v\left(l\right)+\gamma Q\left(x\right(l+1),u(l+1\left)\right)$$

(9)

Therefore, the following formula can be deduced:

$${(\phi \left(l\right)-\gamma \phi (l+1))}^T\Theta \left(\mathbf{H}\right)=v\left(l\right)$$

(10)

There are N equations formed by Formula (10) can be built if $\mathbf{N}$ sets of data on state and control have been collected. Please note that: $\Phi =\left[\begin{array}{c}{({\phi }^1\left(l\right)-\gamma {\phi }^1(l+1))}^T⋮{({\phi }^N\left(l\right)-\gamma {\phi }^N(l+1))}^T\end{array}\right]$

With the least squares method, the solution of the parameters vector $\Theta \left(\mathbf{H}\right)$ can be calculated as following formula when the sufficient excitation condition of system has been satisfied.

$$\Theta \left(\mathbf{H}\right)={({\Phi }^T\Phi )}^{-1}{\Phi }^{T}v$$

(11)

According to the value of $\Theta \left(\mathbf{H}\right)$, the feedback matrix $\mathbf{K}$ can be obtained by Formula (6), and the corresponding control law can be solved.

The complete Q-learning method is an iterative procedure, and it can be fulfilled by the following steps:

• Step 1: An initial feedback matrix ${\mathbf{K}}_0$ is set to control the system;
• Step 2: A set of sufficient excited data are obtained;
• Step 3: The least squares method is used to solve $\Theta \left(\mathbf{H}\right)$;
• Step 4: The new control law is calculated by Formula (6);
• Step 5: The process from step 1 to step 4 will be repeated until $\mathbf{K}$ converges to a stable value.

According to Formula (11), the least squares regression solution has been adopted by the existing Q-learning method to solve $\mathbf{H}$. The Q-learning method for quadratic optimal problem should be a kind of completely or at least partially model-free control. Theoretically, the data sampled from systems are the only needed conditions during the learning procedure instead of the structure or parameters of systems. For linear discrete-time systems, the matrix ${\Phi }^{\mathbf{T}}\Phi$ is singular. Therefore, the least squares regression calculation cannot be implemented.

3.2. Analysis for the Multi-Collinearity of Data Sampled from Linear Discrete-Time Systems

An assumed system defined by Formula (1) has n states and p inputs. Then $\Phi$ of Formula (11) is a matrix composed by n rows and $(n+p)(n+p+1)/2$ columns from n sets of data. When a system with 2 states and 1 input is assumed, the size of matrix $\Phi$ is $l\times 6$ for l sets of data. The ith row of $\Phi$ is expressed as following form:

$${\Phi }^T\left(i\right)=\left[\begin{array}{c}{\left(x_1^i\left(l^i\right)\right)}^2-\gamma {\left(x_1^i(l^i+1)\right)}^2\\ x_1^i\left(l^i\right)x_2^i\left(l^i\right)-\gamma x_1^i(l^i+1)x_2^i(l^i+1)\\ x_1^i\left(l^i\right)u^i\left(l^i\right)-\gamma x_1^i(l^i+1)u^i(l^i+1)\\ {\left(x_2^i\left(l^i\right)\right)}^2-\gamma {\left(x_2^i(l^i+1)\right)}^2\\ x_2^i\left(l^i\right)u^i\left(l^i\right)-\gamma x_2^i(l^i+1)u^i(l^i+1)\\ {\left(u^i\left(l^i\right)\right)}^2-\gamma {\left(u^i(l^i+1)\right)}^2\end{array}\right]$$

(12)

For the quadratic optimal state control problems, there is $u=\mathbf{K}x$. Therefore, Formula (12) can be represented as:

$${\Phi }^T\left(i\right)=\left[\begin{array}{c}{\left(x_1^i\left(l^i\right)\right)}^2-\gamma {\left(x_1^i(l^i+1)\right)}^2\\ x_1^i\left(l^i\right)x_2^i\left(l^i\right)-\gamma x_1^i(l^i+1)x_2^i(l^i+1)\\ k_1[{\left(x_1^i\left(l^i\right)\right)}^2-\gamma {\left(x_1^i(l^i+1)\right)}^2]+k_2[x_1^i\left(l^i\right)x_2^i\left(l^i\right)-\gamma x_1^i(l^i+1)x_2^i(l^i+1)]\\ {\left(x_2^i\left(l^i\right)\right)}^2-\gamma {\left(x_2^i(l^i+1)\right)}^2\\ k_1[x_1^i\left(l^i\right)x_2^i\left(l^i\right)-\gamma x_1^i(l^i+1)x_2^i(l^i+1)]+k_2[{\left(x_1^i\left(l^i\right)\right)}^2-\gamma {\left(x_1^i(l^i+1)\right)}^2]\\ k_1^2[{\left(x_1^i\left(l^i\right)\right)}^2-\gamma {\left(x_1^i(l^i+1)\right)}^2]+2k_{1}k2[x_1^i\left(l^i\right)x_2^i\left(l^i\right)-\gamma x_1^i(l^i+1)x_2^i(l^i+1)]+\cdots k_2^2[{\left(x_2^i\left(l^i\right)\right)}^2-\gamma {\left(x_2^i(l^i+1)\right)}^2]\end{array}\right]$$

(13)

Shown by Formula (13), the third element of row vector $\Phi \left(i\right)$ is a linear combination of the first and the second element; the fifth element is a linear combination of the second and the fourth element; the sixth element is a linear combination of first, the second and the forth element. Therefore, the matrix $\Phi$ is column-related. Generally, for n order linear discrete-time systems, in the $(n+p)(n+p+1)/2$ columns of matrix $\Phi$, the number of linearly independent columns is no more than $n(n+1)/2$. Thus, it is unable to solve the $(n+p)(n+p+1)/2$ parameters directly by the least squares method.

Following corollaries can be deduced by the analysis.

Corollary 1

(Linear discrete-time systems). For the linear discrete-time systems, the quadratic optimal controllers cannot be solved by the Q-learning algorithm adopting the least squares method shown as Formula (11).

Corollary 2

(Linear continuous-time systems). For the linear continuous-time systems described as $\dot{x}=Ax+Bu$, the quadratic optimal controllers cannot be solved by the Q-learning algorithm adopting the least squares method which has the similar form to Formula (11).

For the mentioned continuous systems, the quadratic performance index is: $V\left(x\left(t\right)\right)={\int }_t^{\infty }{e}^{-\gamma (\tau -t)}(x^{T}Qx+u^{T}Ru)dt$. If Q function is defined as:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill Q(x\left(t\right),u\left(t\right))={\int }_t^{t+T}{e}^{-\gamma (\tau -t)}(x^{T}Qx+u^{T}Ru)d\tau +e^{-\gamma T}Q(x(t+T),u(t+T))\\ \hfill =R(x\left(t\right),u\left(t\right))+e^{-\gamma T}Q(x(t+T),u(t+T))\end{array}\end{array}$$

(14)

Therefore, the Q function can be represented as:

$$Q(x\left(t\right),u\left(t\right))={\left[\begin{array}{c}x\left(t\right)\\ u\left(t\right)\end{array}\right]}^T\mathbf{H}\left[\begin{array}{c}x\left(t\right)\\ u\left(t\right)\end{array}\right]={\phi }^T\left(t\right)\Theta \left(\mathbf{H}\right)$$

(15)

where ${\phi }^T\left(t\right)$ and $\Theta \left(\mathbf{H}\right)$ are similar to the definitions in Formula (8). With the state feedback control law $u=\mathbf{K}x$, following equation is insoluble by the existing Q-learning method based on data.

$$(\phi \left(t\right)-e^{-\gamma T}\phi (t+T))\Theta \left(\mathbf{H}\right)=\mathbf{R}(x\left(t\right),u\left(t\right))$$

(16)

3.3. Improved Q-Learning Method Adopting Ridge Regression

To solve the multi-collinearity problem caused by matrix ${\Phi }^T\Phi$, an improved Q-learning method adopting ridge regression has been proposed in this paper. Both ridge regression and generalized inverse are common methods for inversion of singular or ill-conditioned matrices [30,31,32]. According to former studies [32,33], the principles of the two methods are different. With generalized inverse method, the order of model built by matrix ${\Phi }^T\Phi$ will be reduced. In another word, the generalized inverse method is fitter for overparameterization [34] than general irreversible situation. While the ridge regression method maintains the original model order and directly interferes the main diagonal elements of the matrix ${\Phi }^T\Phi$, the regression error will be reduced. Therefore, for the multi-collinearity of matrix ${\Phi }^T\Phi$, the ridge regression is the better solution method.

As an improved least squares method, ridge regression was proposed by Hoerl in 1970. It is an effective supplement for the least squares regression. Ridge regression obtains higher estimate accuracy than the least squares regression, in the meantime, it is a kind of biased estimation method.

The principle of ridge regression is described as follows. There will be $\left|{\Phi }^T\Phi \right|=0$ or $\left|{\Phi }^T\Phi \right|\approx 0$ when $\Phi$ is a rank-deficient matrix. Therefore, the result of Formula (11) will be unsolved or very unstable. A positive matrix $\lambda \mathbf{I}$ is added with ${\Phi }^T\Phi$ to make a new matrix $\left({\Phi }^T\Phi +\lambda \mathbf{I}\right)$, where $\mathbf{I}$ is an unit matrix with the same dimensions as $\Phi$, and $\lambda >0$. As a full rank matrix, $\left({\Phi }^T\Phi +\lambda \mathbf{I}\right)$ is invertible. According to Formula (11), $\Theta \left(\mathbf{H}\right)$ can be solved by following formula when ridge regression is adopted, where $\lambda$ is a minimal positive value.

$$\Theta \left(\mathbf{H}\right)={({\Phi }^T\Phi +\lambda \mathbf{I})}^{-1}{\Phi }^T\nu$$

(17)

The principle of ridge regression is simple and understandable. There are many kinds of selection method for parameter $\lambda$, because the most optimal value of $\lambda$ is dependent on the estimated model. In simulations, the optimized $\lambda$ can be selected by calculating the ridge trace.

The state feedback control law $u=\mathbf{K}x$ can be obtained based on the $\Theta \left(\mathbf{H}\right)$. The improved Q-learning method is still an iterative procedure, which can be realized as following steps.

• Step 1: An initial feedback matrix ${\mathbf{K}}_0$ is given to control the system;
• Step 2: A set of sufficient excited data are obtained;
• Step 3: The $\Theta \left(\mathbf{H}\right)$ is solved based on the ridge regression method Formula (14);
• Step 4: The new control law is calculated by Formula (6);
• Step 5: The process from step 1 to step 4 will be repeated until $\mathbf{K}$ converges to a stable value.

4. Simulations

To make numerical exposition, two simulation examples are shown in following part. Both examples are linear discrete-time system. The simulations show the effectiveness of the improved algorithm.

4.1. Example 1

Example 1: In many optimal control papers the objects are one-order or two-order systems. To show the effectiveness of the proposed method on high order systems, a three-order linear discrete-time system is chosen as following form:

$$\dot{x}=\left[\begin{array}{ccc}1& 0.0997& 0.0045\\ 0& 0.9909& 0.0861\\ 0& -0.1722& 0.7326\end{array}\right]x+\left[\begin{array}{c}0.7502\\ -0.4955\\ 0.0861\end{array}\right]u$$

The weight matrices of performance criteria are: $\mathbf{Q}=\mathbf{I}$, $\mathbf{R}=1$. The discount factor is $\gamma =1$. The initial feedback matrix is set as $\mathbf{K}=\left[\begin{array}{ccc}0.5& 0& 0\end{array}\right]$, and then data can be acquired by solving state equation. The data acquisition process is shown as follows.

To guarantee the sufficiency of excitation, three random numbers will be produced by program as state $x\left(l\right)$ at the lth sample time; $u\left(l\right)$ will be calculated according to values of $x\left(l\right)$; $x(l+1)$ can be derived by state equation, and also the corresponding $u(l+1)$ will be obtained.

For the linear discrete-time system, matrices $\mathbf{A}$ and $\mathbf{B}$ are just used to generate data which is provided for calculation. The feedback matrix $\mathbf{K}$ can be solved without the knowledge of system.

The existing Q-learning calculation process has been calculated based on 200 sets of data sampled from the system. While the step 2 cannot be executed, because of the matrix ${\Phi }^T\Phi$ is irreversible. To avoid the data deficiencies factor, the process is recalculated based on 1500 sets of data, and the same situation appears.

An effective feedback matrix $\mathbf{K}=\left[\begin{array}{ccc}1.0& 0.27& 0.21\end{array}\right]$ can be obtained from the same 200 sets of data based on the improved Q-learning method. Figure 1 shows the closed-loop response and the control effort with the initial state of system $x_0={\left[\begin{array}{ccc}1& 0& 0\end{array}\right]}^T$. In this figure, ordinates are values of states or control variables, and the horizontal ordinates are the computation times of the discrete-time system. The system is stable under the calculated feedback matrix, which means that the effective controller can be solved by the proposed method. There is a distinct different between the solved feedback matrix and the optimal theoretical value, the reason of that is the limitation of the information provided by the sampled data. With more data sets, the calculated feedback matrix will converge to the optimal theoretical value, as shown in example 2.

4.2. Example 2

Example 2: A DC motor discrete-time model described as two-order state equation is:

$$\dot{x}=\left[\begin{array}{cc}1& 0.0952\\ 0& 0.8187\end{array}\right]x+\left[\begin{array}{c}0.0955\\ 0.1813\end{array}\right]u$$

The weight matrices of performance criteria are: $\mathbf{Q}=\mathbf{I}$,$\mathbf{R}=1$. The discount factor is $\gamma =1$.

To compare the optimal theoretical feedback matrix with the calculated one, a controller is designed by the traditional optimal control theory based on the mathematical model. So the matrices $\mathbf{P}$ and ${\mathbf{K}}^{*}$ are solved by Equation (7): $\mathbf{P}=\left[\begin{array}{cc}8.8808& 1.4297\\ 1.4297& 2.9104\end{array}\right]$, ${\mathbf{K}}^{*}=\left[\begin{array}{cc}0.9031& 0.5294\end{array}\right]$. In addition, the solution of Equation (5) is: $\mathbf{H}=\left[\begin{array}{ccc}9.8809& 2.0159& 1.1073\\ 2.0159& 3.2541& 0.6492\\ 1.1073& 0.6492& 1.2262\end{array}\right]$. The $\Theta \left(\mathbf{H}\right)$ can be calculated according to Equation (8): $\Theta \left(\mathbf{H}\right)=\left[\begin{array}{cccccc}9.8809& 4.0319& 2.2146& 3.2541& 1.2984& 1.2262\end{array}\right]$.

To get the feedback matrix by the existing Q-learning method, the initial feedback matrix is set as ${\mathbf{K}}_0=\left[\begin{array}{cc}0.9031& 0.5294\end{array}\right]$. To avoid the data deficiencies factor, the system has been sufficiently excited, and 1500 sets of date is sampled. According to Formula (11), the equation $({\Phi }^T\Phi )\Theta \left(\mathbf{H}\right)={\Phi }^T\nu$ is obtained as following form:

$\left[\begin{array}{cccccc}3.8771& 9.9710& -8.7804& 8.3758& -13.4391& 15.0447\\ 9.9710& 39.8898& -30.1240& 55.0816& -65.1862& 61.7167\\ -8.7804& -30.1240& 23.8783& -36.7265& 46.6489& -46.2619\\ 8.3758& 55.0816& -36.7265& 117.7324& -112.0756& 92.5045\\ -13.4391& -65.1862& 46.6489& -112.0756& 118.2058& -104.7108\\ 15.0447& 61.7167& -46.2619& 92.5045& -104.7108& 97.2166\end{array}\right]\Theta \left(\mathbf{H}\right)=\left[\begin{array}{c}87.3195\\ 362.9179\\ -271.0004\\ 574.5259\\ -631.9216\\ 579.3007\end{array}\right]$

To verify the above equation, the $\mathbf{H}$ calculated by Riccati equation and the $\Theta \left(\mathbf{H}\right)$ are substituted into it, and the two sides of the equation are equal. Therefore, the procedure of the calculation is proved correct. However, the $\Theta \left(\mathbf{H}\right)$ cannot be solved by Equation (15), the analysis is as following. The eigenvalues of ${\Phi }^T\Phi$ is calculated as: $\lambda ({\Phi }^T\Phi )=\left\{\begin{array}{cccccc}0& 0& 0& 0.7968& 26.4209& 373.5823\end{array}\right\}$.

The rank of square matrix ${\Phi }^T\Phi$ is 3, which means that the matrix is singular and noninvertible. Therefore the $\Theta \left(\mathbf{H}\right)$ cannot be calculated via the least squares regression adopted by the existing Q-learning method.

While with the improved Q-learning method, the feedback matrix can be calculated as $\mathbf{K}=\left[\begin{array}{cc}0.5406& 0.0115\end{array}\right]$. According to the ridge trace of $\mathbf{H}$ and $\nu$, the parameter $\lambda$ in Formula (14) is set as 0.01. Figure 1 shows the closed-loop response and the control effects with the initial state of the system as $x_0={\left[\begin{array}{cc}1& 0\end{array}\right]}^T$. Similar to example 1, the Figure 1 shows that the system is closed-loop stable. Like in Figure 2, ordinates of Figure 1 are values of states or control variables, and the horizontal ordinates are the computation times of the discrete-time system. In Figure 3 the ordinates and the horizontal ordinates are the same.

The initial feedback matrix is set as ${\mathbf{K}}_0=\left[\begin{array}{cc}0.5& 0\end{array}\right]$, and the data acquisition process is the same as shown in example 1. The calculation process can be iterated repeatedly until $\mathbf{K}$ converges to a stable value or the system achieves satisfying performance. When the system obtains best performance, the calculated feedback matrix is $\mathbf{K}=\left[\begin{array}{cc}0.9171& 0.4143\end{array}\right]$, and it is very close to the optimal theoretical value ${\mathbf{K}}^{*}=\left[\begin{array}{cc}0.9031& 0.5294\end{array}\right]$. The simulation results are shown in Figure 3. According to the simulation, for the linear discrete-time systems, the quadratic optimal controllers can be designed by online iterative calculation.

5. Further Analysis for Nonlinear Systems

The analysis and the simulation results both present that the quadratic optimal controllers for the linear discrete-time systems cannot be solved by Equation (11). The corresponding analysis should be done to guide the design of Q-learning quadratic optimal controllers for nonlinear discrete-time systems and nonlinear continuous-time systems.

For nonlinear discrete-time systems described as $\mathbf{x}(l+1)=\mathbf{f}\left(\mathbf{x}\right(l),\mathbf{u}(l\left)\right)$, the index of the optimized performance is defined as: $V\left(\mathbf{x}\left(l\right)\right)={\sum }_{k=l}^{\infty }{\gamma }^{k-l}\nu (\mathbf{x}\left(k\right),\mathbf{u}\left(k\right))$. And the Q function is defined as: $Q\left(\mathbf{x}\right(l),\mathbf{u}(l\left)\right)=\nu \left(\mathbf{x}\right(l),\mathbf{u}(l\left)\right)+\gamma Q\left(\mathbf{x}\right(l+1),\mathbf{u}(l+1\left)\right)$. Therefore, the Q function can be noted as:

$$Q(\mathbf{x}\left(l\right),\mathbf{u}\left(l\right))={\phi }^T(\mathbf{x}\left(l\right),\mathbf{u}\left(l\right)){\Theta }_Q$$

(18)

where ${\Theta }_Q$ is the vector of unknown parameters, ${\phi }^T(\mathbf{x}\left(l\right),\mathbf{u}\left(l\right))$ is sampled data set composed by state variables $\mathbf{x}$ and control variables $\mathbf{u}$. The extreme condition derived from Formula (18) is shown as following form: $\frac{\partial Q\left(\mathbf{x}\right(l),\mathbf{u}(l\left)\right)}{\partial \mathbf{u}\left(l\right)}=\frac{\partial {\phi }_Q^T(\mathbf{x}\left(l\right),\mathbf{u}\left(l\right))}{\partial \mathbf{u}\left(l\right)}{\Theta }_Q=0$. The optimized solution of the above equation is noted as:

$$\mathbf{u}\left(l\right)={\phi }_u^T\left(\mathbf{x}\left(l\right)\right){\Theta }_u$$

(19)

where ${\Theta }_u$ is the unknown parameters vector, ${\phi }_u^T\left(\mathbf{x}\left(l\right)\right)$ is the sampled data set composed by state vector $\mathbf{x}$.

Theoretically, the controller can be solved by Formula (18) under the condition derived as Formula (19). While the elements of vector ${\phi }_Q^T(\mathbf{x}\left(l\right),\mathbf{u}\left(l\right))$ should be linear independent to solve ${\Theta }_Q$ adopting the least squares method. So the linear correlation of the vector ${\phi }_Q^T(\mathbf{x}\left(l\right),\mathbf{u}\left(l\right))$ should be discussed to ensure the calculation can be implemented correctly.

For nonlinear continuous-time systems described as: $\dot{\mathbf{x}}=\mathbf{f}(\mathbf{x},\mathbf{u})$, the index of the optimized performance is defined as: $V\left(\mathbf{x}\left(l\right)\right)={\int }_t^{\infty }{e}^{-\gamma (\tau -t)}\nu (\mathbf{x},\mathbf{u})d\tau$. And the Q function is defined as following form.

$$\begin{array}{cc}\hfill l\mathbf{Q}\left(\mathbf{x}\right(l),\mathbf{u}(l\left)\right)& ={\int }_t^{t+T}{e}^{-\gamma (\tau -t)}\nu (\mathbf{x},\mathbf{u})+e^{-\gamma T}\mathbf{Q}(\mathbf{x}(t+T),\mathbf{u}(t+T))\hfill \\ \hfill & =\mathbf{R}(\mathbf{x}\left(l\right),\mathbf{u}\left(l\right))+e^{-\gamma T}\mathbf{Q}(\mathbf{x}(t+T),\mathbf{u}(t+T))\hfill \end{array}$$

(20)

Therefore, the Q function can be noted as:

$$Q(\mathbf{x}\left(t\right),\mathbf{u}\left(t\right))={\phi }_Q^T(\mathbf{x}\left(t\right),\mathbf{u}\left(t\right)){\Theta }_Q$$

(21)

where ${\Theta }_Q$ is the unknown parameters vector, ${\phi }^T(\mathbf{x}\left(t\right),\mathbf{u}\left(t\right))$ is sampled data set composed by state variables and control variables. The extreme condition derived from Formula (20) is shown as the following form: $\frac{\partial Q\left(\mathbf{x}\right(t),\mathbf{u}(t\left)\right)}{\partial \mathbf{u}\left(t\right)}=\frac{\partial {\phi }_Q^T(\mathbf{x}\left(t\right),\mathbf{u}\left(t\right))}{\partial \mathbf{u}\left(t\right)}{\Theta }_Q=0$. The optimized solution of the above equation noted as:

$$\mathbf{u}\left(t\right)={\phi }_u^T\left(\mathbf{x}\left(t\right)\right){\Theta }_u$$

(22)

where ${\Theta }_u$ is the unknown parameters vector, ${\phi }_u^T\left(\mathbf{x}\left(t\right)\right)$ is the sampled data set composed by state vector $\mathbf{x}$.

Similar to nonlinear discrete-time systems, the quadratic optimal controllers for nonlinear continuous-time systems can be solved by Formula (20) under the extreme condition formed as Formula (21). And also the elements of vector ${\phi }_Q^T(\mathbf{x}\left(t\right),\mathbf{u}\left(t\right))$ should be linear independent to solve ${\Theta }_Q$ via the least square method. So the linear correlation of the vector ${\phi }_Q^T(\mathbf{x}\left(t\right),\mathbf{u}\left(t\right))$ should also be discussed to ensure the calculation can be implemented correctly.

In general, the problem about the multi-collinearity of data sets sampled from nonlinear systems no matter discrete-time or continuous-time needs to be discussed in further research to determine whether the existing Q-learning algorithm should be modified when it is used to deal with nonlinear systems. This will make positive significance for nonlinear systems control schemes such as microbial fuel cell systems [35,36]. The relative analysis and simulations will be developed in our further researching.

6. Conclusions

The design of optimal quadratic controllers for linear discrete-time systems based on Q-learning method is claimed as a kind of model-free method [37,38]. In this paper, both the theoretical analysis and the simulation results demonstrate that the existing design method of model-free controllers based on Q-learning algorithm ignores the linear independence of sampled data sets. For the linear discrete-time or continuous-time systems, the model-free quadratic optimal controllers cannot be established directly by the Q-learning method until the linear independence of data sets is guaranteed. The matrix $\Phi$ is inevitably column-related considering the quadratic optimal control law $u=\mathbf{K}x$. So other solution methods should be investigated besides the least squares method to solve parameters matrix $\Theta \left(\mathbf{H}\right)$.

An improved Q-learning method based on ridge regression for linear discrete-time systems is proposed in this paper to deal with the multi-collinearity problem of the sampled data sets. Simulations show the effectiveness of the proposed method. There are deficiencies of the improved Q-learning method. First, the proposed method relies on the ridge trace of the linear correlation matrix, so the computation speed is limited. Second, the proposed method considers with neither disturbance nor saturation of controller. Further research will focus on solving these problems.