A Novel Radial Basis Function Neural Network with High Generalization Performance for Nonlinear Process Modelling

Abstract

A radial basis function neural network (RBFNN), with a strong function approximation ability, was proven to be an effective tool for nonlinear process modeling. However, in many instances, the sample set is limited and the model evaluation error is fixed, which makes it very difficult to construct an optimal network structure to ensure the generalization ability of the established nonlinear process model. To solve this problem, a novel RBFNN with a high generation performance (RBFNN-GP), is proposed in this paper. The proposed RBFNN-GP consists of three contributions. First, a local generalization error bound, introducing the sample mean and variance, is developed to acquire a small error bound to reduce the range of error. Second, the self-organizing structure method, based on a generalization error bound and network sensitivity, is established to obtain a suitable number of neurons to improve the generalization ability. Third, the convergence of this proposed RBFNN-GP is proved theoretically in the case of structure fixation and structure adjustment. Finally, the performance of the proposed RBFNN-GP is compared with some popular algorithms, using two numerical simulations and a practical application. The comparison results verified the effectiveness of RBFNN-GP.

1. Introduction

In recent years, with the continuous development of artificial intelligence and intelligent algorithms, data-driven methods have been widely used as an effective modeling method because they do not require complex mathematical models and high maintenance costs. Among them, the radial basis function neural network (RBFNN) is widely used due to its simple structure and strong nonlinear function approximation ability, especially in the fields of pattern classification, industrial control, nonlinear system modeling and so on [1,2,3,4]. However, there are still some problems to be solved in practice, for example, how to extend the network performance from limited training set to invisible data set, that is, how to design RBFNN with a good generalization ability [5,6]. The generalization performance of RBFNN is usually measured by generalization error, which mainly includes the approximation error caused by the insufficient representation ability of network and estimation errors caused by a limited number of samples. In order to make the RBFNN learnable, the generalization error should be zero as the data tends to infinity.

Due to the limitation of sample data, the network model will produce an estimation error. In order to make the total generalization error close to zero, the number of parameters and samples should tend to infinity to ensure the learnability of the model. Among them, it is worth mentioning that references [5,7,8,9,10] deal with the problem of estimation error according to different assumptions. On this basis, the sample complexity of finite networks is studied to demonstrate that, when the data tends to infinity, the estimation error tends to zero. In addition, due to the limited number of samples, even if the optimal parameter setting is obtained, it will produce functions far from the target, resulting in errors and a poor generalization performance [9]. To solve this problem, Barron et al. [10] introduced the concept of an approximation and estimation bound of artificial neural network, pointing out that, for a kind of common artificial neural network, the integral square error between the estimation network model and the objective function is bounded, and discussing the comprehensive influence of approximation error and estimation error as the objective function on the network accuracy. In addition, Yeung et al. [11] developed a new RBFNN local generalization error model to identify the classifier. By predefining the neighborhood of training samples in the local generalization error model, the upper bound of generalization error of invisible samples was derived. Sarraf [12] proposed a tight upper bound of the generalization error under the assumption of twice continuously differentiable, which was composed of the estimation error under the sample space mean and the expected sensitivity of the error to the input change, and showed how the given upper bound could be used to analyze the generalization error of a feedforward neural network. Although the above methods achieved good results through the generalization error bound based on sensitivity, they still faced challenges due to the computational complexity of partial derivatives, and the generalization error should not only be a function of the number of parameters; it is also important to find a better structure. In addition, Wu et.al. proposed a self-adaptive structural optimal algorithm-based fuzzy neural network (SASOA-FNN) in [13]. This network can improve the generalization ability of the network by minimizing the structural risk model with the number of samples. Terada et al. [14] derived the fast generalization error bound of deep learning under the framework developed in [15]. In the derivation process, they only focused on the minimization of empirical risk and eliminated the scale invariance assumption of activation function. The common feature of the above references is that they are based on risk minimization, and accelerate the convergence speed of the network, while ignoring the influence of the properties of different networks on the generalization error. For example, RBFNN is essentially a local learning method. Each hidden neuron captures the local information of a specific region in the input space by the center and width of its Gaussian activation function [16]. However, the training samples far away from the center of hidden neurons have no effect on the learning of hidden neurons. Therefore, for this local learning method, finding the optimal compromise between model accuracy and generalization error is an effective way to improve the generalization ability of the network.

Different from the estimation error caused by the insufficient samples mentioned above, the approximation error of the network is greatly affected by the network structure. Thus, how to obtain a suitable network structure has always been a hot topic. For instance, Zhou et al. [17] proposed a self-organizing fuzzy neural network with hierarchical pruning scheme (SOFNN-HPS). In SOFNN-HPS, the adaptive ability and robustness of the prediction model were improved through the effective combination of a hierarchical pruning strategy and adaptive allocation strategy. Finally, the accurate prediction of ammonia nitrogen, the key variable in the wastewater treatment process, is realized. To predict the outlet ferrous ion concentration on-line, Xie et al. [18] developed a self-adjusting structure radial basis function neural network (SAS-RBFNN). This algorithm uses the supervised clustering algorithm to initialize the RBFNN, and combines or segments the hidden neurons according to the clustering distance to realize the structural self-organization of RBFNN. In addition, Huang et al. [19] proposed a growing and pruning RBF (GAP-RBF) method based on the significance of a neuron. For the GAP-RBF, the number of neurons can be self-designed to realize a compact RBFNN by linking the significance of neurons and a desired accuracy. On this basis, an improved GAP-RBF algorithm (IGAP-RBF) for any arbitrary distribution of input training data was proposed in reference [20]. This algorithm only adjusts the parameters of the nearest neuron to reduce the computational complexity while ensuring the learning performance. A common feature of GAP-RBF and IGAP-RBF is that the self-organizing strategy is based on the contribution of hidden neurons, and the training samples need to be known in advance. In reference [21], an adaptive gradient multi-objective particle swarm optimization algorithm was proposed to predict the biochemical oxygen demand, a key water quality parameter in the wastewater treatment process. This method adopts a multi-objective gradient method and adaptive flight parameter mechanism, which not only greatly reduces computational complexity, but also improves generalization performance; other structure self-organization methods are outlined in [22,23,24,25]. The advantage of the above algorithms is that they can adjust the network parameters while adjusting the network structure, and the disadvantage is that different parameter adjustment methods make the learning speed of the network different, which may affect the accuracy of the network.

Most of the existing methods focus on using self-organizing strategies to obtain appropriate structures, or using effective learning algorithms to obtain a higher accuracy. However, good training accuracy is not equal to good generalization performance. Therefore, designing an effective learning model to improve the generalization performance of RBFNN is still an urgent problem that needs to be solved. Based on the above analysis, a self-organizing RBFNN based on network sensitivity is proposed to improve the generalization performance. The main contributions of this method are as follows.

The generalization ability is quantified by network sensitivity. Then, an RBFNN-GP algorithm is constructed to improve the generalization performance:

1.

The convergence of the RBFNN-GP is verified in theory, which ensures its successful application;

2.

The effectiveness and feasibility of the RBFNN-GP are verified by predicting the key water quality parameters in wastewater treatment process.

The remainder of this paper is organized as follows. Section 2 briefly introduces the basic RBFNN and the local generalization error bound of the network. Then, the details of RBFNN-GP are given in Section 3. The convergence of RBFNN-GP is discussed in Section 4. Section 5 presents the experimental results of RBFNN-GP to demonstrate its advantages. The application field and future work direction of the proposed method are shown in Section 6. Finally, the conclusions are given in Section 7.

2. Materials and Methods

2.1. Radial Basis Function Neural Network (RBFNN)

In general, RBFNN consists of three layers: the input layer, the hidden layer and the output layer. A typical multiple-input, single-output RBFNN (MISO-RBFNN) is shown in Figure 1. The MISO-RBFNN is a k-m-1 network, and each neuron in the RBFNN hidden layer is constructed in the form of Gaussian function. The mathematical description of RBFNN output is as follows:

$$\widehat{y}\left(t\right)={\displaystyle \sum _{j=1}^m{w}_j\left(t\right){\theta }_j\left(t\right)},$$

(1)

where m is the number of hidden neurons, w_j(t) is the weight between the jth hidden layer and the output layer at time t, and ${\theta }_j\left(t\right)$ is the output of the jth hidden layer neuron, described as:

$${\theta }_j\left(t\right)=e^{-\frac{{‖x\left(t\right)-c_j\left(t\right)‖}^2}{2{\sigma }_j^2\left(t\right)}},$$

(2)

$$c_j\left(t\right)={\left[c_{1,j}\left(t\right),c_{2,j}\left(t\right),\cdots ,c_{n,j}\left(t\right)\right]}^{{\rm T}},$$

(3)

where x(t) = [x₁(t), x₂(t), …, x_n(t)]^T is the input vector, c_j(t) is the center vector of the jth hidden neuron at time t, and n is the dimension of the input vector; ||x(t)−c_j(t)|| is the Euclidean distance between x(t) and c_j(t), and σ_j(t) is the width of the jth hidden neuron at time t.

2.2. Local Generalization Error Bound

The generalization error of the whole input space is defined as [11]:

$$E_{gen}(t)={\displaystyle \underset{D}{\int }{\left[f\left(x(t)\right)-F\left(x(t)\right)\right]}^2}p\left(x(t)\right)dx(t),$$

(4)

where x(t) is the input vector in the input space and p(x(t)) is the unknown probability density function of x(t). Given a training data set $D=\left(x_a\left(t\right),F_a\left(t\right)\right), a=1,2,\cdots ,N$, N is the number of pairs of input and output, namely the number of samples of training set. The empirical error of network can be defined as:

$$E_{emp}\left(t\right)=\frac{1}{N}{\displaystyle \sum _{a=1}^N{\left[\widehat{f}\left(x_a\left(t\right)\right)-F\left(x_a\left(t\right)\right)\right]}^2},$$

(5)

where $\widehat{f}\left(x_a\left(t\right)\right)$ and $F\left(x_a\left(t\right)\right)$represent the approximate and real mapping functions between the ath input and output in the input space, respectively. The ultimate goal of improving the generalization ability is minimize the approximation error, and the network can directly predict the unseen data.

Since RBFNN is a local method, for each sample $x_a\left(t\right)\in D$, we can find a sample set:

$$X_{S,x_a}\left(t\right)=\left\{\left.x\left(t\right)\right|x\left(t\right)=x_a\left(t\right)+\Delta x\left(t\right)\right\} ,$$

(6)

where $X_{S,x_a}\left(t\right)$ defines an S -neighborhood of the training sample $x_a\left(t\right)$, $\Delta x\left(t\right)=[\Delta x_1\left(t\right),\cdots ,\Delta x_n\left(t\right)]$ is regarded as perturbations, n denotes the number of input features, and S is a given number. The samples in $X_{S,x_a}\left(t\right)$ (except $x_a\left(t\right)$) are regarded as unseen samples. For $0\le S_1\le \cdots \le S_k\le \infty$, the relationship holds$D\subseteq X_{S_1}\left(t\right)\subseteq \cdots \subseteq X_{S_k}\left(t\right)\subseteq I$, where I is the entire input space. By Hoeffding’s inequality, we can derive the definition of local generalization error bound as follows [11,26]:

$$E_{gen,S}\left(t\right)=\sqrt{E_{emp}\left(t\right)}+\sqrt{E_{X_S,\Delta y^2}\left(t\right)} ,$$

(7)

with:

$$\begin{array}{l}\Delta y\left(t\right)=\widehat{f}\left(x_a\left(t\right)\right)-F\left(x_a\left(t\right)\right)\\ E_{X_s,\Delta y^2}\left(t\right)=\frac{1}{N}{\displaystyle \sum _{a=1}^N{\displaystyle {\int }_{X_s\left(x_a\left(t\right)\right)}\left(\Delta y^2\left(t\right)\right)\frac{1}{{\left(2S\right)}^a}dx\left(t\right)},}\end{array}$$

(8)

where $\Delta y\left(t\right)$ is the difference between the network output and the real value; the term $E_{X_S,\Delta y^2}\left(t\right)$ is introduced in Section 3.1.

Remark 1.

Different from previous methods, to the best of our knowledge, this work is a new attempt to obtain looser generalization error bounds by eliminating high-order terms and reducing partial accuracy in exchange for better generalization ability.

3. RBFNN with High Generation Performance (RBFNN-GP)

The proposed RBFNN-GP, which can improve the network generalization ability, is introduced in this section. It contains the following two parts: (1) sensitivity measurement (SM) method and (2) structural self-organization optimization (SSO) strategy.

3.1. Sensitivity Measurement (SM) Method

Sensitivity analysis provides an effective method to evaluate the impact of different inputs on output results, and it can accurately express the causal response between input changes and corresponding outputs [27,28]. Different from existing studies that focus on improving modeling accuracy or looking for indicator variables, in this study, SM is introduced to quantify the impact of network input changes on output changes, and to intuitively represent the sensitivity of network output to input changes. Thus, the network structure can be adjusted accordingly. Suppose that the inputs are independent and not identically distributed; then, from this, each input feature has its own expectation ${\mu }_{x_i}$ and variance, ${\delta }_{x_i}^2$:

$${\varphi }_j\left(t\right)=w_j^2\left(t\right)e^{{\delta }_{d_j}\left(t\right)/2{\sigma }_j^4\left(t\right)-{\mu }_{d_j}\left(t\right)/{\sigma }_j^2\left(t\right)},$$

(9)

with:

$$\left\{\begin{array}{l}{d}_j\left(t\right)={‖x\left(t\right)-c_j\left(t\right)‖}^2\\ {\delta }_{d_j}\left(t\right)={\displaystyle {\sum }_{i=1}^n\mu \left[{\left(x_i\left(t\right)-{\mu }_{x_i}\left(t\right)\right)}^4\right]}-{\left({\delta }_{x_i}^2\left(t\right)\right)}^2+4{\delta }_{x_i}^2\left(t\right){\left({\mu }_{x_i}\left(t\right)-c_{ji}\left(t\right)\right)}^2 +\\ 4\mu \left[{\left(x_i\left(t\right)-{\mu }_{x_i}\left(t\right)\right)}^3\right]\left({\mu }_{x_i}\left(t\right)-c_{ji}\left(t\right)\right)\end{array}\right.$$

(10)

and:

$$\left\{\begin{array}{l}{\mu }_{d_j}\left(t\right)={\displaystyle {\sum }_{i=1}^n\left[{\delta }_{x_i}^2\left(t\right)+{\left({\mu }_{x_i}\left(t\right)-c_{ji}\left(t\right)\right)}^2\right]}\\ {\sigma }_j\left(t\right)={\varphi }_j\left(t\right){\displaystyle {\sum }_{i=1}^n\left({\delta }_{x_i}^2\left(t\right)+{\left({\mu }_{x_i}\left(t\right)-c_{ji}\left(t\right)\right)}^2/{\sigma }_j^4\left(t\right)\right)}\end{array}\right..$$

(11)

In theory, as long as the input variation is finite, we do not strictly limit its data distribution. In this instance, we assume that the unseen samples S -neighborhood of the training samples obey the uniform distribution, and thus we obtain ${\delta }_{\Delta x_i}^2\left(t\right)=S^2/3$. By the law of large numbers, the sensitivity of RBFNN is:

$$\begin{array}{l}{E}_X{}_{{}_S,\Delta y^2}\left(t\right)\\ =\frac{1}{N}\left\{{\displaystyle {\sum }_{a=1}^N{\displaystyle {\int }_{X_{S,x_a}}{\left[f\left(x_a\left(t\right)+\Delta x\left(t\right)\right)-f\left(x_a\left(t\right)\right)\right]}^{2}p\left(\Delta x\left(t\right)\right)}} d\Delta x\left(t\right)\right\}\\ \approx {\displaystyle {\sum }_{j=1}^m{\varphi }_j\left(t\right)\left\{{\displaystyle {\sum }_{i=1}^n\left[{\delta }_{\Delta x_i}^2\left(t\right)\left({\delta }_{x_i}^2\left(t\right)+{\left({\mu }_{x_i}\left(t\right)-c_{ji}\left(t\right)\right)}^2+0.2{\delta }_{\Delta x_i}^2\left(t\right)\right)\right]}/{\sigma }_j^4\left(t\right)\right\}}\end{array}$$

(12)

with:

$${\varphi }_j\left(t\right)={\sigma }_j^4\left(t\right){\xi }_j\left(t\right).$$

(13)

therefore, we can obtain:

$$E_X{}_{{}_S,\Delta y^2}\left(t\right)\approx \frac{1}{45}{S}^{4}n{\displaystyle {\sum }_{j=1}^m{\xi }_j\left(t\right)+\frac{1}{3}{S}^2{\displaystyle {\sum }_{j=1}^m{\sigma }_j\left(t\right)}} .$$

(14)

Remark 2.

Limiting cases of $E_{gen,S}\left(t\right)$. Clearly, when$S\to \infty , X_S\left(t\right)\to I,$that is$S\to 0, X_S\left(t\right)\to D$. If$0\le S_1\le \cdots \le S_k\le S_{\infty }$, the relationship$D\subseteq X_{S_1}\left(t\right)\subseteq \cdots \subseteq X_{S_k}\left(t\right)\subseteq I$ holds. Therefore, in the case of $S\to \infty$, $E_{gen}\left(t\right)<E_{gen,S}\left(t\right)$.

Remark 3.

Statistical performance. Compared with the regression error bound, which only uses the effective parameters and the number of training samples, the proposed RBFNN-GP algorithm has clear advantages, because it considers statistical characteristics, such as the mean and variance of the training data set.

3.2. Structural Self-Organization Optimization (SSO) Strategy

In order to construct a RBFNN with a high generalization performance, an SSO strategy is designed based on the sensitivity measurement. This SSO strategy can adjust the structure and parameters (including center, width and weight) of RBFNN at the same time. The self-organization strategies are shown as follows:

$$\left\{\begin{array}{l}c\left(t+1\right)=c\left(t\right)-\eta \Delta c\left(t\right)\\ \sigma \left(t+1\right)=\sigma \left(t\right)-\eta \Delta \sigma \left(t\right)\\ w\left(t+1\right)=w\left(t\right)-\eta \Delta w\left(t\right)\\ m\left(t+1\right)=\left\{\begin{array}{l}m\left(t\right)+1, {\sigma }_j<{\lambda }_1\\ m\left(t\right), {\lambda }_1<{\sigma }_j<{\lambda }_2\\ m\left(t\right)-1, {\sigma }_j>{\lambda }_2\end{array}\right.\end{array}\right.,$$

(15)

where ${\sigma }_j\ge 1$, c(t + 1) = [c₁(t + 1), c₂(t + 1),…, c_m(t + 1)] and $\sigma \left(t+1\right)=[{\mathsf{\sigma }}_1\left(t+1\right),{\mathsf{\sigma }}_2\left(t+1\right),\dots ,{\mathsf{\sigma }}_m\left(t+1\right)]$ are the center and width vectors of the hidden neuron at time t + 1, w(t + 1) is the weight of output layer at time t + 1, m(t) represents the number of hidden layer neurons at time t, and $m\left(t\right)\ge 1$, $\eta$ is the learning rate, ${\lambda }_1\le -2/3S^{2}n{\xi }_m$, ${\xi }_m$ is the ratio of statistical output and width of the mth neuron (${\lambda }_1$ is a dynamic threshold and the statistical residual is negative), and ${\lambda }_2\ge 0$ are used to ensure the convergence performance of the network (here the value is twice that of the input dimension). It should be noted that, only when ${\lambda }_1$ and ${\lambda }_2$ acquire the equals sign at the same time, will the number of neurons remain unchanged, that is, the structure of the neural network holds. The variables of $\Delta c\left(t\right)$, $\Delta \sigma \left(t\right)$, $\Delta w\left(t\right)$ present the changes of the centers, widths and weights at time t, respectively. We obtain:

$$\left\{\begin{array}{l}\Delta c\left(t\right)=\left[\Delta c_1\left(t\right),\Delta c_2\left(t\right),\cdots ,\Delta c_m\left(t\right)\right]\\ \Delta \sigma \left(t\right)=\left[\Delta {\sigma }_1\left(t\right),\Delta {\sigma }_2\left(t\right),\cdots ,\Delta {\sigma }_m\left(t\right)\right]\\ \Delta w\left(t\right)=\left[\Delta w_1\left(t\right),\Delta w_2\left(t\right),\cdots ,\Delta w_m\left(t\right)\right]\end{array}\right.,$$

(16)

where $\Delta c_m\left(t\right)=\left[\Delta c_{m,1}\left(t\right),\cdots ,\Delta c_{mn}\left(t\right)\right]$ is the change of the center of the mth neuron at time t, and $\Delta {\sigma }_m\left(t\right)$ and $\Delta w_m\left(t\right)$ are the changes of width and weight of the mth neuron at time t, respectively. Moreover, the updates details of the parameters are:

$$\left\{\begin{array}{l}\Delta c_j\left(t\right)=\partial E_{emp}\left(t\right)/\partial c_j\left(t\right)=\left(x\left(t\right)-c_j\left(t\right)/{\sigma }_j^2\right)w_j{\theta }_j\left(1-{\theta }_j\right)e\left(t\right)\\ \Delta {\sigma }_j\left(t\right)=\partial E_{emp}\left(t\right)/\partial {\sigma }_j\left(t\right)=\left[{\left(x\left(t\right)-c_j\left(t\right)\right)}^2/{\sigma }_j^3\right]w_j{\theta }_j\left(1-{\theta }_j\right)e\left(t\right)\\ \Delta w_j\left(t\right)=\partial E_{emp}\left(t\right)/\partial w_j\left(t\right)={\theta }_{j}e\left(t\right)\end{array}\right..$$

(17)

Based on the above analysis, the detailed steps of neuron growth and pruning are given below.

3.2.1. Growth Stage

If ${\sigma }_j<{\lambda }_1$, new neurons are added to the neural network to reduce the approximation error and improve the generalization performance. At this time, the number of neurons becomes, and the parameter of new neurons is:

$$\left\{\begin{array}{l}{c}_{new}=x\left(t\right) \\ {\sigma }_{new}=\frac{1}{m}{\displaystyle \sum _{j=1}^m{\sigma }_i\left(t\right)}\\ w_{new}=\left(y\left(t\right)-\widehat{y}\left(t\right)\right)e^{{\displaystyle \sum _{i=1}^n\frac{{\left(x_i\left(t\right)-c_{i,new}\left(t\right)\right)}^2}{2{\sigma }_{i,new}^2\left(t\right)}}}\end{array}\right.,$$

(18)

where $c_{new}$, ${\sigma }_{new}$ and $w_{new}$ represent the center, width and weight of the new neuron,$x_i\left(t\right)$represents the ith element in the input vector at time t, $c_{i,new}\left(t\right)$ and ${\sigma }_{i,new}\left(t\right)$ are the ith elements of the center and width of the new neuron at time t, respectively. After structural adjustment, the parameters are updated as:

$$\left\{\begin{array}{l}c\left(t+1\right)=\left[c\left(t\right);c_{new}\right]\\ \sigma \left(t+1\right)=\left[\sigma \left(t\right);{\sigma }_{new}\right]\\ w\left(t+1\right)=\left[w\left(t\right);w_{new}\right]\end{array}\right.,$$

(19)

where $c\left(t+1\right)$, $\sigma \left(t+1\right)$ and $w\left(t+1\right)$ are the center, width and weight of RBFNN at time t + 1, respectively.

3.2.2. Prune Stage

If ${\sigma }_j>{\lambda }_2$, the jth neuron with the least information in the hidden layer is deleted. In this way, the network complexity is reduced under the premise of ensuring generalization ability. At this time, the number of neurons becomes m(t − 1), and the parameters of the new neurons are:

$$\left\{\begin{array}{l}{c}_j\left(t+1\right)=0\\ {\sigma }_j\left(t+1\right)=0\\ w_j\left(t+1\right)=0\end{array}\right.,$$

(20)

where $c_j\left(t+1\right)$, ${\sigma }_j\left(t+1\right)$ and $w_j\left(t+1\right)$ represent the center, width and weight of the jth neuron at time t + 1, respectively. After structural adjustment, the parameters of the ith neuron were as follows:

$$\left\{\begin{array}{l}{c}_i\left(t+1\right)=c_i\left(t\right)\\ {\sigma }_i\left(t+1\right)={\sigma }_i\left(t\right)\\ w_i\left(t+1\right)=w_i\left(t\right)+\frac{w_j{\theta }_j\left(t\right)}{{\theta }_i\left(t\right)}\end{array}\right..$$

(21)

Among them, the ith neuron is the neuron closest to the Euclidean distance from the jth neuron, $c_i\left(t+1\right)$, ${\sigma }_i\left(t+1\right)$ and $w_i\left(t+1\right)$ are the center, width and weight of the ith neuron at time t + 1.

Remark 4.

Because there is only one hidden layer in the model, the network structure is simple, which greatly reduces the error accumulation in the process of back propagation. Furthermore, no extra parameters are added during network training, which reduces the amount of computation. Therefore, the stability of the network is well guaranteed.

4. Convergence Analysis

Another important problem of neural network structure is convergence analysis, which not only affects the application in practical engineering, but also reflects the generalization ability of neural network. If the neural network cannot guarantee convergence or meet convergence conditions, it is difficult to realize the successful application of the neural network. In addition, for RBFNN-GP, its convergence is not only related to the parameter optimization algorithm, but also related to structural changes. Therefore, this paper analyzes the convergence from three aspects: convergence in the stable stage, growth stage and deletion stage.

Hypothesis 1 (H1).

The center c of the hidden layer, the width $\sigma$ of the hidden layer and the input–output weight w satisfy the boundedness, that is $‖c‖\le {\mu }_c,‖\sigma ‖\le {\mu }_{\sigma },‖w‖\le {\mu }_w$, where ${\mu }_c$,${\mu }_{\sigma }$,${\mu }_w$ are positive real numbers.

Hypothesis 2 (H2).

There is a set of “optimal” network parameters, $c^{*},{\sigma }^{*}$ and $w^{*}$, that is, the optimal center, width and weight.

4.1. Convergence Analysis of RBFNN with Fixed Structure

For the convenience of discussing its convergence, the differential equation of e is expressed as follows [29].

$$\dot{e}\left(t\right)=-e\left(t\right)+w^{*{\rm T}}\left(t\right){\theta }^{*}\left(t\right)-w^{{\rm T}}\left(t\right)\theta \left(t\right),$$

(22)

where $\Delta \theta ={\theta }^{*}-\theta ,\Delta w=w^{*}-w$, $\Delta \theta$ is the change in the hidden layer neuron output, $\Delta w$ is the change of weight. Equation (22) is reformulated as:

$$\dot{e}\left(t\right)=-e\left(t\right)+w^{*{\rm T}}\left(t\right)\Delta \theta \left(t\right)+\Delta w^{{\rm T}}\left(t\right)\theta \left(t\right).$$

(23)

In order to transform the nonlinear output of the network into a partially linear form, the Taylor expansion of $\Delta \theta$ is:

$$\Delta \theta ={\phi }_{{}_c}^{{\rm T}}\left(c^{*}-c\right)+{\phi }_{{}_{\sigma }}^{{\rm T}}\left({\sigma }^{*}-\sigma \right)+\mathrm{\Omega },$$

(24)

where $\mathrm{\Omega }$ is the higher order infinitesimal of Taylor expansion.

Theorem 1.

Suppose the number of hidden-layer neurons of RBFNN is fixed, and the network parameters are updated according to Equtions (14)–(16); when $t\to \infty , e\left(t\right)\to 0$, the convergence of the network is guaranteed.

Proof of Theorem 1. The Lyapunov function is defined as:

$$V\left(e,c,\sigma ,w\right)=\frac{1}{2}\left(e^2+\Delta c^{{\rm T}}\Delta c+\Delta {\sigma }^{{\rm T}}\Delta \sigma +\Delta w^{{\rm T}}\Delta w+\Delta v^{{\rm T}}\Delta v\right)$$

(25)

with:

$$\left\{\begin{array}{l}\Delta c=c^{*}-c\\ \Delta \sigma ={\sigma }^{*}-\sigma \\ \Delta w=w^{*}-w\\ \dot{v}=e\\ \Delta v=v^{*}-v=w^{*}{}^{{\rm T}}\left({\phi }_{{}_c}^{{\rm T}}{c}^{*}+{\phi }_{\sigma }^{{\rm T}}{\sigma }^{*}+\mathrm{\Omega }\right)-w^{{\rm T}}\left({\phi }_{{}_c}^{{\rm T}}{c}^{*}+{\phi }_{\sigma }^{{\rm T}}{\sigma }^{*}\right)\end{array}\right.$$

(26)

where v is the compensator and $v^{*}$ is the optimal compensator. The partial derivative of the Lyapunov function is:

$$V^{\prime }\left(e,c,\sigma ,w\right)=e{e}^{\prime }+\Delta {c^{\prime }}^{{\rm T}}\Delta c+\Delta {{\sigma }^{\prime }}^{{\rm T}}\Delta \sigma +\Delta {w^{\prime }}^{{\rm T}}\Delta w+\Delta {v^{\prime }}^{{\rm T}}\Delta v.$$

(27)

According to Equations (22)–(26), we can obtain:

$$\begin{array}{l}{V}^{\prime }\left(e,c,\sigma ,w\right)\\ =e{e}^{\prime }+{\left(c^{*}-c\right)}^{\prime }{}^{{\rm T}}\Delta c+{\left({\sigma }^{*}-\sigma \right)}^{\prime }{}^{{\rm T}}\Delta \sigma +{\left(w^{*}-w\right)}^{\prime }{}^{{\rm T}}\Delta w+{\left(v^{*}-v\right)}^{\prime }{}^{{\rm T}}\Delta v\\ =-e^2+e\left\{\begin{array}{l}{w}^{*{\rm T}}\left[{\phi }_{{}_c}^{{\rm T}}\left(c^{*}-c\right)+{\phi }_{{}_{\sigma }}^{{\rm T}}\left({\sigma }^{*}-\sigma \right)+\mathrm{\Omega }\right]+\Delta w^{{\rm T}}\theta -\\ w^{{\rm T}}\left[{\phi }_{{}_c}^{{\rm T}}\left(c^{*}-c\right)+{\phi }_{{}_{\sigma }}^{{\rm T}}\left({\sigma }^{*}-\sigma \right)\right]-{\phi }_w^{{\rm T}}\Delta w\end{array}\right\}-e\left(v^{*}-v\right)\\ =-e^2+e\left[v^{*}-v+\Delta w^{{\rm T}}\left(-{\phi }_{{}_c}^{{\rm T}}c-{\phi }_{{}_{\sigma }}^{{\rm T}}\sigma +\theta \right)-{\phi }_w^{{\rm T}}\Delta w\right]-e\left(v^{*}-v\right)\\ =-e^2\end{array}$$

(28)

and:

$$V^{\prime }\left(e,c,\sigma ,w\right)\le 0.$$

(29)

Thus,$V^{\prime }$ is the seminegative definite in the above space. In light of the Lyapunov theorem, we can obtain:

$$\underset{t\to \infty }{\lim }e\left(t\right)=0.$$

(30)

So far, the convergence of a fixed-structure RBFNN is proved. □

4.2. Convergence Analysis of RBFNN with Changeable Structure

Theorem 2.

If the network structure is self-organizing in the learning process, then the parameters are adjusted according to Equations (14)–(21). When$t\to \infty , e\left(t\right)\to 0$, the convergence of RBFNN based on sensitivity can be guaranteed.

Proof Theorem 2. The structure self-organization process of RBFNN is divided into two parts: structure growth and structure pruning stages. □

4.2.1. Growth Stage

At time t, there are m neurons in the hidden layer of self-organizing RBFNN based on sensitivity, and the network error is e_m(t). When the growth condition is satisfied, the number of hidden layer neurons is increased by 1. Then, the number of hidden layer neurons is m + 1, and the output error of the RBFNN is:

$$e_{m+1}\left(t\right)=\frac{1}{2}{\left(\widehat{y}\left(t\right)-{\widehat{y}}_{m+1}\left(t\right)\right)}^2,$$

(31)

where e_m+1(t) is the network error of m + 1 hidden neurons, ${\widehat{y}}_m\left(t\right)$ and ${\widehat{y}}_{m+1}\left(t\right)$ represent the network output before and after the addition of hidden layer neurons, respectively. According to Equations (17)–(19), we can obtain:

$$e_{m+1}\left(t\right)=\frac{1}{2}{\left[\widehat{y}\left(t\right)-\left({\displaystyle \sum _{j=1}^m{w}_j\left(t\right){\theta }_j\left(t\right)+w_{m+1}\left(t\right){\theta }_{m+1}\left(t\right)}\right)\right]}^2=0.$$

(32)

so:

$$e_{m+1}\left(t\right)=0.$$

(33)

It can be seen that when a new neuron is added to the hidden layer, the convergence speed of the network is accelerated based on the parameter setting of the newly added neurons.

4.2.2. Prune Stage

When the pruning condition is satisfied, the jth neuron in the hidden layer is deleted, and the error of the network is e_m−1(t). Thus, the error of the network can be rewritten as:

$$e_{m-1}\left(t\right)={\widehat{y}}_m\left(t\right)-\left({\displaystyle \sum _{l=1}^m{w}_l{\theta }_l\left(t\right)}-w_j{\theta }_j\left(t\right)\right),$$

(34)

where ${\widehat{y}}_m\left(t\right)$represents the network output when the number of hidden layer neurons is m:

$$\begin{array}{ll}{e}_{m-1}\left(t\right)& ={\widehat{y}}_m\left(t\right)-\left({\displaystyle \sum _{l=1,l\ne i}^m{w}_l{\theta }_l\left(t\right)} -w_j{\theta }_j\left(t\right)+\left(w_i+w_j\frac{{\theta }_j\left(t\right)}{{\theta }_i\left(t\right)}\right){\theta }_i\left(t\right)\right)\\ & ={\widehat{y}}_m\left(t\right)-\left({\displaystyle \sum _{l=1,l\ne i}^m{w}_l{\theta }_l\left(t\right)} +w_i{\theta }_i\left(t\right)\right)\\ & =0\end{array}$$

(35)

In summary, the error of the neural network remains unchanged before and after the jth neuron is deleted, that is, the process of deleting neurons does not destroy the convergence of the original neural network.

5. Experimental Studies

This section describes the experiments conducted to assess the effectiveness of the generalization performance of the RBFNN-GP algorithm. The experiment includes one benchmark and two practical problems, namely, the approximation of the Mexican straw hat function and the prediction of key water quality parameters, ammonia nitrogen and membrane permeability in the wastewater treatment process. In addition, the good generalization performance of the proposed RBFNN-GP is further illustrated by comparisons with the existing six algorithms.

5.1. Benchmark Example A

In this case, the RBFNN-GP algorithm is applied to approximate the Mexican straw hat function, which is a benchmark problem used in [30,31] to checkout many prevalent algorithms. The Mexican straw hat function is:

$$y=\sin (\sqrt{x_1^2+x_2^2})/\sqrt{x_1^2+x_2^2}.$$

(36)

In the training phase, the training samples are $x={\left\{x_1,x_2;y\right\}}^N$, where x₁, x₂ are stochastic and generated within the scope of (−2π, 2π), N = 2100 is the number of training samples; the testing samples are {x₁, x₂, y}^M, where M = 700 is the number of testing samples, and the learning rate is set to η = 0.003.

The experimental results are shown in Figure 2, where Figure 2a shows the number of hidden neurons in the training process. It can be seen that the proposed RBFNN-GP can adjust the network structure by pruning or increasing the number of neurons in the learning process. Figure 2b shows the change trajectory of the root-mean-square error (RMSE) in the learning process. Meanwhile, the prediction error results and output error surface are depicted in Figure 2c,d. It is clear that the proposed RBFNN-GP can approximate the Mexican straw hat function with small predicting errors. In order to further prove the excellent generalization ability of the proposed method, the prediction results of the RBFNN-GP are compared with those of the other dynamic neural networks based on structural adjustment, such as SASOA-FNN [12], SOFNN-HPS [17], SAS-RBFNN [18], AGMOPSO [21], ASOL-SORBFNN [23] and the RBFNN with a fixed structure (fixed-RBFNN). In order to make the comparison more meaningful, all algorithms in this experiment use the same data set, including training samples and test samples, and ensure that the initial number of neurons is the same. In addition, all of the algorithms run 10 times, and then take the average value to make the results more convincing. The results are shown in

Table 1. Comparison results with other self-organizing methods.

Methods	No. of NNs	CPU Time(s)		Testing RMSE			Training RMSE
		Mean	Dev.	Mean	Dev.	Max.	Mean
Fixed-RBFNN	8	100.10	0.096	0.031	0.0037	0.040	0.042
SASOA-FNN [12]	8	108.29	0.031	0.029	0.0043	0.033	0.041
SOFNN-HPS [17]	12	131.02	0.076	0.047	0.0053	0.063	0.057
SAS-RBFNN [18]	9	119.35	0.024	0.039	0.0051	0.059	0.052
AGMOPSO [21]	8	106.29	0.028	0.024	0.0037	0.039	0.041
ASOL-SORBFNN [23]	8	135.46	0.031	0.035	0.0046	0.041	0.040
RBFNN-GP	7	100.36	0.012	0.027	0.0033	0.029	0.039

, where Max. is the maximum and Dev. is the deviation.

As can be seen from Table 1, the proposed RBFNN-GP requires fewer hidden nodes and output errors and has a better generalization ability than the self-organizing network based on information minimization and structural risk minimization. This mainly depends on the fact that the method considers not only the number of effective parameters in the network, but also the mean and variance of input data.

5.2. Benchmark Example B

In this example, the effectiveness of the RBFNN-GP is applied for the Mackey–Glass chaotic time series prediction problem, which is a famous benchmark [32,33]. The discrete expression of the time series is given by:

$$\begin{array}{l}x\left(t+1\right)=\left(1-a_1\right)x\left(t\right)+a_{2}x\left(t-\tau \right)/\left[1+x^{10}\left(t-\tau \right)\right]\\ x\left(t+1\right)=f\left[x\left(t\right),x\left(t-6\right),x\left(t-12\right),x\left(t-18\right)\right]\end{array}$$

(37)

where a₁, a₂, τ are constants, x(0) represents the initial value, and a₁ = 0.1, a₂ = 0.2, τ = 17, x(0) = 1.2. In the training phase, 1000 data samples are extracted from t = 21 to 1021. Additionally, 500 samples are used as training samples and 500 samples as test samples. The preset training error is 0.001, the initial learning rate ŋ = 0.02, and the number of neurons is 5. It should be noted that the other parameters in all comparison methods are the same, and the running results of the experiment are shown in Figure 3.

From Figure 3a, it can be seen that there are several increasing and decreasing stages in the learning process of RBFNN-GP. In the early period of training, the number of neurons changes frequently, and the network structure is unstable. Figure 3b shows the comparison between the predicted output of RBFNN-GP and the real value. Figure 3c,d show the prediction error value of the network and the RSME value in the training stage, respectively. It is worth mentioning that the reason why the RSME curve is so smooth is closely related to the sensitivity of the network. Since the mean and variance of the training samples are fully considered, the stability of the network is improved.

Similarly,

Table 2. Comparison results with other self-organizing methods.

Methods	No. of NNs	CPU Time(s)		Testing RMSE			Training RMSE
		Mean	Dev.	Mean	Dev.	Max.	Mean
Fixed-RBFNN	8	104.10	0.069	0.036	0.0039	0.035	0.312
SASOA-FNN [12]	7	126.29	0.030	0.029	0.0041	0.028	0.124
SOFNN-HPS [17]	11	116.3	0.076	0.054	0.0066	0.068	0.261
SAS-RBFNN [18]	10	123.97	0.026	0.034	0.0045	0.031	0.219
AGMOPSO [21]	8	119.89	0.029	0.027	0.0039	0.031	0.217
ASOL-SORBFNN [23]	8	133.91	0.065	0.043	0.0044	0.040	0.251
RBFNN-GP	6	105.36	0.022	0.026	0.0032	0.019	0.215

shows the comparative experimental results with the other four methods: SASOA-FNN [12], SOFNN-HPS [17], SAS-RBFNN [18], AGMOPSO [21], ASOL-SORBFNN [23] and fixed-RBFNN. As shown in Table 2, the optimal number of hidden nodes obtained by using the RBFNN-GP is only six, and the mean and deviation of the test error are minimal in the comparison method. Since the parameters are updated at the same time in the process of structural adjustment to avoid repeated calculations, the computational complexity is greatly reduced compared with the dynamic structural adjustment method based on information strength. Nevertheless, compared with other self-organizing networks based on structural risk, the proposed RBFNN-GP takes a little longer time to calculate the mean and variance information of input samples. However, we still have reason to believe that RBFNN-GP demonstrates a good compromise between generalization ability and training accuracy.

5.3. Permeability Prediction of Membrane Bio-Reactor

Membrane bio-reactor (MBR) is a new wastewater treatment technology combining membrane separation technology and biotechnology, which is widely used in wastewater treatment process (WWTP). However, in the process of MBR wastewater treatment, membrane pollution will shorten the service life of the membrane and cause an unnecessary loss of process energy consumption. It is of great practical significance to correctly predict the permeability of the membrane and increase its working efficiency [34,35,36]. Therefore, the proposed RBFNN-GP is applied to predict the permeability of MBR in WWTP. The real data of the experiment come from a wastewater treatment plant in Beijing, China. After disposing of the abnormal data, 500 samples were obtained and normalized.

In this experiment, 50 training samples and 50 test samples are selected to test the performance of RBFNN-GP. In order to remove the correlation between variables, partial least squares method was used to select nine variables from twenty-two variables as the input of RBFNN-GP. Due to the wide range of membrane bioreactor permeability, the number of iterations is T = 200, the learning rate is η = 0.003, and the time length is h = 10. The intuitive prediction results are shown in Figure 4. Figure 4a shows the comparison results between the actual and predicted values of membrane bioreactor permeability, and the prediction error is shown in Figure 4b. It can be seen from Figure 4 that the proposed RBFNN-GP has a good prediction performance for the permeability of the MBR, and the prediction error within the range [−4, 3].

Moreover,

Table 3. Comparison results with other self-organizing methods.

Methods	No. of NNs	CPU Time(s)		Testing RMSE			Training RMSE
		Mean	Dev.	Mean	Dev.	Max.	Mean
Fixed-RBFNN	8	105.65	0.037	0.033	0.0034	0.037	0.031
SASOA-FNN [12]	8	109.21	0.035	0.028	0.0033	0.021	0.024
SOFNN-HPS [17]	11	126.34	0.032	0.042	0.0049	0.025	0.032
SAS-RBFNN [18]	9	121.62	0.042	0.025	0.0038	0.029	0.035
AGMOPSO [21]	8	114.25	0.031	0.030	0.0042	0.031	0.114
ASOL-SORBFNN [23]	11	125.31	0.040	0.028	0.0041	0.040	0.127
RBFNN-GP	6	108.24	0.032	0.025	0.0040	0.022	0.022

shows the comparison results of SASOA-FNN [12], SOFNN-HPS [17], SAS-RBFNN [18], AGMOPSO [21], ASOL-SORBFNN [23] and Fixed-RBFNN in predicting the permeability of MBR. It can be seen from Table 3 that, under the same iteration times, the learning time of SASOA-FNN [12] and RBFNN-GP is almost the same. However, the accuracy of the latter is better than that of the former. Thus, we have sufficient reasons to believe that the RBFNN-GP method shows a great improvement in training accuracy, calculation speed and generalization ability compared with the above methods.

6. Discussion

6.1. Computational Complexity

Computational complexity is an important indicator for evaluating the model. For the proposed RBFNN-GP, the calculation involved is closely related to the training process of and N. Suppose $\left[x(t),y(t);t=1,\cdots ,N\right]$ is a set of training samples. When the structure of RBFNN-GP is k-m-l (k represents the input variable, m represents the number of hidden layer neurons, and l represents the output variable), the computational complexity of RBFNN-GP is O[m(t)]. It can be seen that the computational burden of RBFNN-GP is not heavy.

6.2. Future Trends

In this paper, which aims to realize the online prediction of key water quality parameter membrane pollution in wastewater treatment process, a prediction model based on self-organizing RBFNN is established that meets the needs of accurate prediction of key water quality parameters in wastewater treatment process. At the same time, the self-organizing network modeling method is developed. Due to its good generalization performance and theoretical support, the proposed method can also be extended to other types of networks, such as multi-input, single-output fuzzy neural networks and multi-input, multiple-output RBF neural network.

7. Conclusions

In this paper, an RBFNN-GP algorithm is proposed to improve the model generalization ability. Firstly, the upper bound of the local generalization error is found, and the network structure and parameters are adjusted within the allowable error range. Then, a generalization error formula based on network sensitivity and approximate error is introduced to improve the generalization performance, while ensuring its accuracy. Finally, experimental validation is carried out on two different benchmark data sets and a real application of wastewater treatment process. The results show that the RBFNN-GP algorithm can learn robust networks with good generalization performance and compact scale. In summary, RBFNN-GP has the following advantages:

1.

With the help of sensitivity measurements and locally generalized error bounds, the network has a statistical performance and can reasonably achieve structure self-organization without a high dependence on sample numbers.

2.

The convergence of RBFNN-GP for fixed and variable structures is guaranteed by the thresholds λ₁ and λ₂. Therefore, the proposed RBFNN-GP can not only reduce the number of additional parameters, but also decreases the computational burden.

3.

Compared with existing algorithms, the proposed RBFNN-GP shows a good generalization ability in the prediction of key water quality parameters in wastewater treatment processes. Furthermore, this approach can be extended to other types of networks and industrial domains.