Locating the Parameters of RBF Networks Using a Hybrid Particle Swarm Optimization Method

Abstract

In the present work, an innovative two-phase method is presented for parameter tuning in radial basis function artificial neural networks. These kinds of machine learning models find application in many scientific fields in classification problems or in function regression. In the first phase, a technique based on particle swarm optimization is performed to locate a promising interval of values for the network parameters. Particle swarm optimization was used as it is a highly reliable method for global optimization problems, and in addition, it is one of the fastest and most-flexible techniques of its class. In the second phase, the network was trained within the optimal interval using a global optimization technique such as a genetic algorithm. Furthermore, in order to speed up the training of the network and due to the use of a two-stage method, parallel programming techniques were utilized. The new method was applied to a number of famous classification and regression datasets, and the results were more than promising.

1. Introduction

Regression and data classification are two major categories of problems that are solved with machine learning techniques. Such problems appear regularly in scientific fields such as physics [1,2], chemistry [3,4], economics [5,6], medicine [7,8], etc. A programming tool that is used quite often to handle such problems is the Radial Basis Function (RBF) artificial neural network [9]. An RBF network can be defined as the following function:

$$y\left(\overrightarrow{x}\right)=\sum _{i=1}^k{w}_i\varphi \left(∥\overrightarrow{x}-\overrightarrow{c_i}∥\right)$$

(1)

The following applies to the above equation:

• The vector $\overrightarrow{x}$ stands for the input pattern to the Equation (1). The number of elements in this vector is denoted as d.
• The vectors $\overrightarrow{c_i},i=1,..,k$ are denoted as the center vectors.
• The vector $\overrightarrow{w}$ is considered as the output weight of the RBF network.
• The value $y\left(\overrightarrow{x}\right)$ represents the predicted value of the network for the pattern $\overrightarrow{x}$.

Typically, the Gaussian function can bed used as the function $\varphi \left(x\right)$, and it is defined as:

$$\varphi \left(x\right)=exp\left(-\frac{{\left(x-c\right)}^2}{{\sigma }^2}\right)$$

(2)

A plot of the previous function with $c=0,\sigma =1$ is displayed in Figure 1. As can be observed, the value of the function decreases as we move away from the center. An extensive overview of RBF networks was given in the work of Ghosh and Nag [10]. RBF networks are used as approximation tools in various cases, such as solutions to differential equations [11,12], digital communications [13,14], physics [15,16], chemistry [17,18], economics [19,20,21], network security [22,23], etc. RBF networks were thoroughly discussed in [24], and they have been parallelized in a variety of research papers [25,26]. This model has been extended by various researchers in tasks such as creating new initialization techniques for the network parameters, [27,28,29], pruning techniques [30,31,32], the construction of RBF networks [33,34,35] etc.

In this work, a hybrid technique is proposed for the optimal calculation of the parameters of an RBF network. This technique consists of two phases. During the first phase, information was collected from the training data of the neural network and an attempt was made to identify a small interval of values for the neural network parameters. To identify this interval, an optimization method was used, which gradually creates the optimal value interval, which was estimated to give the lowest value for the training error of the network. To locate the optimal interval, the Particle Swarm Optimization (PSO) technique was used [36]. The PSO method was chosen for the first phase because it is fast and flexible enough for optimization problems, does not require a large number of parameters to be input by the user, and has been successfully used in a variety of problems such as flow shop scheduling [37], developing charging strategies for electric vehicles [38], emotion recognition [39], robot trajectory planning [40], etc. The detection of the value interval was performed in order to then make the minimization of the network error faster and more efficient in the second phase of the optimization method. In the second phase, the parameters of the neural network were optimized within the optimal value interval of the first phase. The optimization can be performed by any global optimization method [41]. In this work, genetic algorithms [42,43,44] were chosen for the second phase. The main advantages of genetic algorithms are tolerance to errors, easy implementation in parallel, efficient exploration of the search space, etc.

Recently, much work has been appeared to tune the parameters of machine learning models, such as the work of Agarwal and Bhanot [45] for the adaptation of the RBF parameters, the incorporation of an improved ABC algorithm to adapt the parameters of RBF networks [46], the usage of the Firefly algorithm for optimization [47], along with machine learning models for cervical cancer diagnosis [48], the adaptation of the CNN and XGBOOST models by an optimization algorithm for COVID-19 diagnosis [49], etc.

The rest of this article is organized as follows: in Section 2, the two phases of the proposed method are thoroughly discussed; in Section 3, the experimental datasets are listed, as well as the experimental results; finally, in Section 4, some conclusions are presented.

2. Method Description

The training error of the RBF network is expressed as:

$$E\left(y(x,g)\right)=\sum _{i=1}^m{\left(y\left(x_i,g\right)-t_i\right)}^2$$

(3)

The value m stands for the number of patterns, and the values $t_i$ denote the real output for the input. The vector g denotes the set of parameters of the RBF network. Usually, RBF networks are trained through a two-phase procedure:

• In the first phase, the k centers, as well as the associated variances are calculated through the K-means algorithm [50]. A typical formulation of the K-means algorithm is outlined in Algorithm 1.
• In the second phase, the weight vector $\overrightarrow{w}=\left(w_1,w_2,\dots ,w_k\right)$ is estimated by solving a linear system of equations:

  (a)

  Set $W=w_{kj}$;

  (b)

  Set $\Phi ={\varphi }_j\left(x_i\right)$;

  (c)

  Set $T=\left\{t_i=f\left(x_i\right),i=1,..,M\right\}$;

  (d)

  The system to be solved is identified as:

  $${\Phi }^T\left(T-\Phi W^T\right)=0$$

  (4)

  with the solution:

  $$W^T={\left({\Phi }^T\Phi \right)}^{-1}{\Phi }^{T}T={\Phi }^{†}T.$$

  (5)

The proposed work used two computational phases to optimally calculate the network parameters. Firstly, a promising range of the parameters of the network was calculated through an optimization process that incorporated interval arithmetic. Subsequently, the parameters of the network were trained with the usage of a genetic algorithm inside the located range of the first phase. The following subsections analyze both of these phases in detail.

Algorithm 1 The K-means algorithm.

Repeat. (a) $S_j=\left\{\right\},j=1..k$. (b) For each pattern $x_i,i=1,...,m$, do: i. Calculate$j^{\ast }={min}_{i=1}^k\left\{D\left(x_i,c_j\right)\right\}$. ii. Update$S_{j^{\ast }}=S_{j^{\ast }}\cup \left\{x_i\right\}$. (c) EndFor. (d) For each center $c_j,j=1..k$, do: i. Define$M_j$ as the number of points in $S_j$. ii. Calculate$c_j$: $$c_j=\frac{1}{M_j}\sum _{i=1}^{M_j}{x}_i$$ (e) EndFor. Compute the variances for every center as $${\sigma }_j^2=\frac{{\sum }_{i=1}^{M_j}{\left(x_i-c_j\right)}^2}{M_j}$$ Terminate if there is no change in centers $c_j$.

2.1. Preliminaries

In order to perform interval arithmetic on RBF networks, the following definitions are introduced:

• The comparison of two intervals $W=\left[w_1,w_2\right]$, $Z=\left[z_1,z_2\right]$ is performed through the function:

  $$\begin{array}{ccc}\hfill L^{\ast }(W,Z)& =& \left\{\begin{array}{cc}TRUE,\hfill & w_1<z_1,OR\left(w_1=z_{1}AND{w}_2<z_2\right)\hfill \\ FALSE,\hfill & OTHERWISE;\hfill \end{array}\right.\hfill \end{array}$$

  (6)
• The function $E\left(y\right)$ (Equation (3)) is modified to an interval one $\left[E_{\text{min}}\left(y\right),E_{\text{max}}\left(y\right)\right]$ calculated with the procedure given in Algorithm 2.

In the proposed algorithm, the RBF network contains n variables, where

$$n=(d+2)\times k$$

(7)

The value of n is calculated as follows:

• Every center $\overrightarrow{c_i},i=1,..,k$ has d variables, which means $d\times k$ variables.
• For every center, a separate value ${\sigma }_i$ is used for the Gaussian processing unit, which means k variables.
• The output weight $\overrightarrow{w}$ also has k variables.

Algorithm 2 Fitness calculation for the modified PSO algorithm.

The fitness calculation for a given particle g is as follows: Take$N_S$ random samples in g. Calculate$E_{min}\left(g\right)={min}_{g_i\in N_S}\left({\left({\sum }_{j=1}^{M}y\left(x_j,g_i\right)-t_j\right)}^2\right)$. Calculate$E_{max}\left(g\right)={max}_{g_i\in N_S}\left({\left({\sum }_{j=1}^{M}y\left(x_j,g_i\right)-y_j\right)}^2\right)$. Return$f_g=\left[E_{min}\left(g\right),E_{max}\left(g\right)\right].$

2.2. The Proposed PSO Algorithm

During this phase, arithmetic interval techniques are used to locate a range for the parameters of the RBF network. The interval techniques [51,52,53] comprise a common method in global optimization with various applications [54,55,56]. The first phase aims to locate the most-promising bounding box for the n parameters of the corresponding RBF network. The initial bounding box is defined as S, which is a subset of ${\mathbb{R}}^{\mathbb{n}}$:

$$S=\left[a_1,b_1\right]\otimes \left[a_2,b_2\right]\otimes \dots \left[a_n,b_n\right]$$

(8)

The interval method of the first phase divides the set S subsequently by discarding areas that are not promising enough to contain the global minimum. In order to locate the best interval for Equation (8), a modified PSO algorithm [57] is used. The proposed variant of the PSO method is based on the original technique (Algorithm 1 of [57]); however, the particles are intervals of values, and at each iteration, a normalization of the velocity vector takes place to avoid generating particles outside the original range of values. The PSO method is based on a population of candidate solutions, which, in most cases, are called particles. The method is based on two vectors: the current location of particles denoted as $\overrightarrow{p}$ and the velocity of their movement denoted as $\overrightarrow{u}$. The PSO method finds the global minimum by moving the particles based on their previous best position, as well as the best position of the total population of particles.

The initial bounding boxes for the centers and variances of the RBF network are constructed using the K-means clustering algorithm. Subsequently, the initial values for the intervals $\left[a_i,b_i\right]$ are calculated through Algorithm 3. The values for the intervals of the first $(d+1)\times k$ variables are obtained as a multiple of the positive quantity F with the values obtained by K-means. The value B is used to initialize the intervals for the output weight $\overrightarrow{w}$. Afterwards, the following PSO variant is executed:

• Set$N_c$ as the amount of particles.
• Set the normalization factor $\lambda$.
• Set the k weights of the RBF network.
• Set$N_g$ the maximum generations allowed.
• Set$N_s$ the number of random samples that will be used in the fitness calculation algorithm.
• Set$f^{\ast }=\left[\infty ,\infty \right]$, the fitness of the best located particle $p^{\ast }$.
• Construct$S=\left[a_1,b_1\right]\otimes \left[a_2,b_2\right]\otimes \dots \left[a_n,b_n\right]$, as obtained from the previous two algorithms.
• Initialize the $N_c$ particles. Each particle $p_i,i=1,...,N_c$ is considered as a set of intervals randomly initialized in S. The layout of each particle is graphically presented in Figure 2.
• For$i=1,...,N_c$, do:

  (a)

  Calculate the fitness $f_i$ of particle $p_i$ using the procedure outlined in Algorithm 2.

  (b)

  If$L^{\ast }\left(f_i,f^{\ast }\right)=\mathrm{T}\mathrm{R}\mathrm{U}\mathrm{E}$, then$f^{\ast }=f_i,p^{\ast }=p_i$

  (c)

  Set$p_{b,i}=p_i,f_{b,i}=f_i$ as the best located position for particle i and the associated fitness value.

  (d)

  For$j=1,...,n$, do:

  i.

  Set$\delta$ the width of interval $p_{ij}$.

  ii.

  Set$u_{ij}=\left[-r\frac{\delta }{20},r\frac{\delta }{20}\right]$, with r a random number in $[0,1]$. The velocity is initialized to a small sub-interval of the range of values for the corresponding parameter in order to avoid, as much as possible, excessive values for the velocity. This would result in the particles moving out of their value range very quickly, thus making the optimization process difficult.

  (e)

  EndFor.

• EndFor.
• Set iter = 0.
• Calculate the inertia value as $\omega ={\omega }_{\text{max}}-\frac{\text{iter}}{N_g}\left({\omega }_{\text{max}}-{\omega }_{\text{min}}\right)$, where common values for these parameters are ${\omega }_{\text{min}}=0.4$ and ${\omega }_{\text{max}}=0.9$. Many inertia calculations have appeared in the relevant literature such as constant inertia [58], linearly decreasing inertia [59], exponential inertia [60], random inertia calculation [61], dynamic inertia [62], fuzzy inertia calculation [63], etc. The present method of calculating the inertia was chosen because it decreases linearly with time, and for large values of the inertia, it allows a wider search in the search space, while for low values, it allows a more focused search.
• For$i=1,...,N_c$, do:

  (a)

  Calculate the new velocity $u_i=\omega u_i+r_1{c}_1\left(p_{b,i}-p_i\right)+r_2{c}_2\left(p^{\ast }-p_i\right)$, where $r_1,r_2$ are random numbers in $[0,1]$, and the constant values $c_1$ and $c_2$ stand for the cognitive and the social parameters, correspondingly. Usually, the values for $c_1$ and $c_2$ are in $[1,2]$.

  (b)

  Normalize the velocity as: $u_i=\frac{1}{\lambda }{u}_i$, where $\lambda$ is a positive number with $\lambda >1$.

  (c)

  Update the position $p_i=p_i+u_i$.

  (d)

  Calculate the fitness $f_i$ of particle $p_i$.

  (e)

  If$L^{\ast }\left(f_i,f_{b,i}\right)=\text{TRUE}$, then$p_{b,i}=p_i,f_{b,i}=f_i$.

  (f)

  If$L^{\ast }\left(f_i,f^{\ast }\right)=\text{TRUE}$, then$f^{\ast }=f_i,p^{\ast }=p_i$.

• EndFor.
• Set iter = iter+1.
• If$\text{iter}\le N_g$, goto Step 13.
• Else, return$S=\left[a_1,b_1\right]\otimes \left[a_2,b_2\right]\otimes \dots \left[a_n,b_n\right]$, the domain range for the best particle $p^{\ast }$.

Algorithm 3 Algorithm used to locate the initial values for $\left[a_i,b_i\right],i=1,...,n$.

Set m = 0. Set$F>1,B>0$. For$i=1..k$, do: (a) For$j=1..d$, do: i. Set$a_m$ = $-F\times c_{ij}$, $b_m$ = $F\times c_{ij}$. ii. Set$m=m+1$. (b) EndFor. (c) Set$a_m=-F\times {\sigma }_i$, $b_m=F\times {\sigma }_i$. (d) Set$m=m+1$. EndFor. For$j=1,...,k$, do: (a) Set$a_m=-B,b_m=B$. (b) Set$m=m+1$. EndFor.

2.3. Optimization of Parameters through Genetic Algorithm

During the second phase of the proposed method, a genetic algorithm is implemented, which optimizes the parameters of the RBF network within the optimal interval calculated in the first phase. The used genetic algorithm has its roots in the $\mathrm{G}\mathrm{A}\left(c_{r1},l\right)$ algorithm from the paper of Kaelo and Ali [64]. This method was enhanced using the stopping rule suggested by Tsoulos [65]. This genetic algorithm has the following steps:

• Initialization step:

  (a)

  Set$N_c$ as the number of chromosomes. Every chromosome is coded as in the case of PSO using the scheme of Figure 2.

  (b)

  Set$N_g$ as the maximum number of generations allowed.

  (c)

  Setk as the weight number of the RBF network.

  (d)

  Obtain the domain range S from the procedure of Section 2.2.

  (e)

  Initialize$N_C$ randomly in S.

  (f)

  Define the selection rate $p_s\in [0,1]$.

  (g)

  Define the mutation rate $p_m\in [0,1]$.

  (h)

  Set iter = 0.

• Evaluation step:
  For every chromosome g, calculate the associated fitness value $f_g={\sum }_{i=1}^m{\left(y\left(x_i,g\right)-t_i\right)}^2$:
• Genetic operations step:
  Perform the genetic operations of selection, crossover, and mutation.

  (a)

  Selection procedure: First, the population of chromosomes is sorted based on the associated fitness values. The first $\left(1-p_s\right)\times N_c$ chromosomes are copied unchanged to the next generation, while the rest are replaced by offspring constructed by the crossover procedure. During the selection step, a series of mating pairs is chosen using the well-known procedure of tournament selection for each parent.

  (b)

  Crossover procedure: For each pair $(z,w)$ of chosen parents, two new offspring $\tilde{z}$ and $\tilde{w}$ are constructed with the steps:

  $$\begin{array}{ccc}\hfill \widetilde{z_i}& =& a_i{z}_i+\left(1-a_i\right)w_i\hfill \\ \hfill \widetilde{w_i}& =& a_i{w}_i+\left(1-a_i\right)z_i\hfill \end{array}$$

  (9)

  where $a_i$ is a random number with $a_i\in [-0.5,1.5]$ [64].

  (c)

  Mutation procedure: For every element of each chromosome, pick a random number $r\in [0,1]$. If$r\le p_m$, then alter randomly the corresponding element.

• Termination check step:

  (a)

  Set$iter=iter+1$.

  (b)

  If the termination criteria hold, then Terminate; else, goto evaluation step.

The overall process of the two phases is graphically shown in Figure 3.

3. Experiments

The suggested method was tested on a series of classification and regression problems found from various papers and sites of the relevant literature. For the classification problems, two Internet databases were used:

• The UCI dataset repository, https://archive.ics.uci.edu/ml/index.php (accessed on 5 January 2023).
• The Keel repository, https://sci2s.ugr.es/keel/datasets.php (accessed on 5 January 2023) [66].

The regression problems can be found at the Statlib URL ftp://lib.stat.cmu.edu/datasets/index.html (accessed on 5 January 2023).

3.1. Experimental Datasets

The classification problems used here were the following:

• Appendicitis dataset, a medical dataset suggested in [67].
• Australian dataset [68], an economic dataset.
• Balance dataset [69], used for the prediction of psychological states.
• Cleveland dataset, related to heart diseases [70,71].
• Bands dataset, a dataset related to printing problems [72].
• Dermatology dataset [73], which is a medical dataset.
• Hayes-roth dataset [74].
• Heart dataset [75], a medical dataset.
• HouseVotes dataset [76].
• Ionosphere dataset, a dataset from the Johns Hopkins database [77,78].
• Liverdisorder dataset [79], a medical dataset about liver disorders.
• Lymography dataset [80].
• Mammographic dataset [81], which is a dataset about breast cancer.
• Parkinsons dataset, a medical dataset about Parkinson’s Disease (PD) [82].
• Pima dataset, a medical dataset [83].
• Popfailures dataset [84], a dataset about climate.
• Spiral dataset: The spiral artificial dataset contains 1000 two-dimensional examples that belong to two classes (500 examples each). The number of features is 2. The data in the first class were created using the following formula: $x_1=0.5tcos\left(0.08t\right),x_2=0.5tcos\left(0.08t+\frac{\pi }{2}\right)$, and the second class data using: $x_1=0.5tcos\left(0.08t+\pi \right),x_2=0.5tcos\left(0.08t+\frac{3\pi }{2}\right)$.
• Regions2 dataset, described in [85].
• Saheart dataset [86], which is related to heart diseases.
• Segment dataset [87], which is related to image processing.
• Wdbc dataset [88], which is related to breast tumors.
• Wine dataset. The wine recognition dataset contains data from wine chemical analysis. It contains 178 examples of 13 features each, which are classified into three classes. It has been examined in many published works [89,90].
• Eeg dataset. As a real-word example, an EEG dataset described in [91] was used here. The datasets derived from the dataset are denoted as Z_F_S, ZONF_S, and ZO_NF_S.
• Zoo dataset [92], used for the classification of animals.

The regression datasets were as follows:

• Abalone dataset [93].
• Airfoil dataset, a dataset from NASA related to aerodynamic and acoustic tests [94].
• Baseball dataset, a dataset used to predict the points scored by baseball players.
• BK dataset [95], used to estimate the points scored per minute in a basketball game.
• BL dataset; this dataset is related to an experiment on the affects of machine adjustments on the time to count bolts.
• Concrete dataset, related to civil engineering [96].
• Dee dataset, used to predict the daily average price of electric energy in Spain.
• Diabetes dataset, a medical dataset.
• FA dataset, related to fat measurements.
• Housing dataset, described in [97].
• MB dataset, a statistics dataset [95].
• MORTGAGE dataset, which contains economic data.
• NT dataset, derived from [98].
• PY dataset (the Pyrimidines problem) [99].
• Quake dataset, which contains data from earthquakes [100].
• Treasure dataset, which contains economic data.
• Wankara dataset, which is about weather measurement

3.2. Experimental Results

The RBF network for the tests was coded in ANSI C++ with the help of the freely available Armadillo library [101]. In addition, in order to have greater reliability of the experimental results, a 10-fold validation technique was used. All the experiments were executed 30 times with different seeds for the random generator each time, and the average was measured. For the classification datasets, the average classification error is reported and, for the regression datasets, the mean test error. The machine used for the experiments was an AMD Ryzen 5950X with 128GB of RAM. The used operating system was Debian Linux. In order to accelerate the training process, the OpenMP library was incorporated [102]. The experimental settings are listed in

Table 1. The used values for the experimental parameters. The first column denotes the name of the parameter and the second the used value.

Parameter	Value
$N_c$	200
$N_g$	100
$N_s$	50
$c_1$	1.0
$c_2$	1.0
F	5.0
B	100.0
k	10
$p_s$	0.90
$p_m$	0.05

. The experimental results for the classification datasets are listed in

Table 2. Experimental results for the classification datasets. The first column is the name of the used dataset.

Dataset	NN-RPROP	NN-GENETIC	RBF-KMEANS	IRBF-100	IRBF-1000
Appendicitis	16.30%	18.10%	12.23%	16.47%	14.03%
Australian	36.12%	32.21%	34.89%	23.61%	22.39%
Balance	8.81%	8.97%	33.42%	12.65%	13.15%
Bands	36.32%	35.75%	37.22%	37.38%	36.29%
Cleveland	61.41%	51.60%	67.10%	49.77%	49.64%
Dermatology	15.12%	30.58%	62.34%	38.24%	35.64%
Hayes Roth	37.46%	56.18%	64.36%	33.62%	34.13%
Heart	30.51%	28.34%	31.20%	15.91%	15.60%
HouseVotes	6.04%	6.62%	6.13%	4.77%	3.90%
Ionosphere	13.65%	15.14%	16.22%	8.64%	7.52%
Liverdisorder	40.26%	31.11%	30.84%	27.36%	25.63%
Lymography	24.67%	23.26%	25.31%	19.12%	20.02%
Mammographic	18.46%	19.88%	21.38%	17.17%	17.30%
Parkinsons	22.28%	18.05%	17.41%	15.51%	13.59%
Pima	34.27%	32.19%	25.78%	23.61%	23.23%
Popfailures	4.81%	5.94%	7.04%	5.21%	5.10%
Regions2	27.53%	29.39%	38.29%	26.08%	25.77%
Saheart	34.90%	34.86%	32.19%	27.94%	28.91%
Segment	52.14%	57.72%	59.68%	47.19%	40.28%
Spiral	46.59%	44.50%	44.87%	19.43%	19.56%
Wdbc	21.57%	8.56%	7.27%	5.33%	5.44%
Wine	30.73%	19.20%	31.41%	9.20%	6.84%
Z_F_S	29.28%	10.73%	13.16%	4.19%	4.18%
ZO_NF_S	6.43%	8.41%	9.02%	4.31%	4.35%
ZONF_S	27.27%	2.60%	4.03%	2.23%	2.08%
ZOO	15.47%	16.67%	21.93%	10.13%	11.13%
AVERAGE	26.86%(3)	24.87%(1)	29.03%(1)	19.43%(8)	18.68%(13)

and, for the regression datasets, in

Table 3. Experimental results for the regression datasets. The first column is the name of the used regression dataset.

DATASET	NN-RPROP	NN-GENETIC	RBF-KMEANS	IRBF-100	IRBF-1000
ABALONE	4.55	7.17	7.37	5.57	5.32
AIRFOIL	0.002	0.003	0.27	0.004	0.003
BASEBALL	92.05	103.60	93.02	78.89	85.58
BK	1.60	0.03	0.02	0.04	0.03
BL	4.38	5.74	0.013	0.0003	0.0003
CONCRETE	0.009	0.009	0.011	0.007	0.007
DEE	0.608	1.013	0.17	0.16	0.16
DIABETES	1.11	19.86	0.49	0.78	0.89
HOUSING	74.38	43.26	57.68	20.27	21.54
FA	0.14	1.95	0.015	0.032	0.029
MB	0.55	3.39	2.16	0.12	0.09
MORTGAGE	9.19	2.41	1.45	0.39	0.78
NT	0.04	0.006	8.14	0.007	0.007
PY	0.039	1.41	0.012	0.024	0.014
QUAKE	0.041	0.040	0.07	0.04	0.03
TREASURY	10.88	2.93	2.02	0.33	0.51
WANKARA	0.0003	0.012	0.001	0.002	0.002
AVERAGE	11.71(1)	11.34(1)	10.17(5)	6.27(7)	6.76(3)

. For the experimental tables, the following were applied:

• The column NN-PROP indicates the application of the Rprop method [103] in an artificial neural network [104,105] with 10 hidden nodes. The RPROP method is coded in the FCNN software package [106].
• The column NN-GENETIC denotes the application of a genetic algorithm in the artificial neural network with 10 hidden nodes. The parameters of the used genetic algorithm are the same as in the second phase of the proposed method.
• The column RBF-KMEANS denotes the classic training method for RBF networks by estimating centers and variances through K-means and the output weights by solving a linear system of equations.
• The column IRBF-100 denotes the application of the current method with $\lambda =100$.
• The column IRBF-1000 denotes the application of the current method with $\lambda =1000$.
• In both tables, an extra line is added, in which the mean error for each method is shown. This row is denoted by the name AVERAGE. This line also shows the number of times the corresponding method achieved the best result. This number is shown in parentheses.

As one can see from the experimental results, the proposed method significantly outperformed the other techniques in the majority of cases in terms of the average error in the test set. Moreover, the difference from the established method of training RBF networks was of the order of 40%, and in some cases, this percentage can be doubled. The statistical difference of the proposed technique against the rest is also shown in Figure 4 and Figure 5. However, the proposed technique was significantly slower than the original training technique, as it is a two-stage technique. In the first stage, an optimal interval of values for the network parameters is created with a modified PSO method, and in the second stage, the network is trained using a genetic algorithm. Of course, this extra time can be significantly reduced by incorporating parallel techniques, as was done experimentally using the OpenMP library. Furthermore, changing the normalization factor $\lambda$ from 100 to 1000 did not have much effect on the mean error in the test set. This implies that the proposed method is quite robust, since it does not have much dependence on this parameter.

An additional experiment was performed with different values for the parameter F. The experimental results for this experiment are shown in

Table 4. Experimental results with the proposed method and using different values for the parameter F on the classification datasets.

DATASET	$\mathit{F}=3$	$\mathit{F}=5$	$\mathit{F}=10$
Appendicitis	14.43%	14.03%	14.47%
Australian	23.45%	22.39%	23.21%
Balance	13.35%	13.15%	11.79%
Bands	36.48%	36.29%	36.76%
Cleveland	49.26%	49.64%	49.02%
Dermatology	36.54%	35.64%	34.37%
Hayes Roth	39.28%	34.13%	36.46%
Heart	15.14%	15.60%	14.89%
HouseVotes	4.93%	3.90%	6.41%
Ionosphere	7.56%	7.52%	9.05%
Liverdisorder	28.37%	25.63%	28.97%
Lymography	20.12%	20.02%	21.05%
Mammographic	18.04%	17.30%	18.21%
Parkinsons	18.51%	13.59%	13.49%
Pima	23.69%	23.23%	23.52%
Popfailures	5.76%	5.10%	4.50%
Regions2	25.79%	25.77%	25.32%
Saheart	28.89%	28.91%	26.99%
Segment	36.53%	40.28%	43.28%
Spiral	16.78%	19.56%	22.18%
Wdbc	4.64%	5.44%	5.10%
Wine	8.31%	6.84%	8.27%
Z_F_S	4.32%	4.18%	4.03%
ZO_NF_S	3.70%	4.35%	3.72%
ZONF_S	2.04%	2.08%	1.98%
ZOO	11.87%	11.13%	9.97%
AVERAGE	18.65%	18.68%	19.12%

for the classification datasets and in

Table 5. Experimental results with the proposed method using different values for the parameter F on the classification datasets.

DATASET	$\mathit{F}=3$	$\mathit{F}=5$	$\mathit{F}=10$
ABALONE	5.56	5.32	5.41
AIRFOIL	0.004	0.003	0.004
BASEBALL	88.40	85.58	84.43
BK	0.03	0.03	0.02
BL	0.0005	0.0003	0.0002
CONCRETE	0.009	0.007	0.007
DEE	0.18	0.16	0.16
DIABETES	0.67	0.89	0.77
HOUSING	20.03	21.54	20.84
FA	0.03	0.029	0.036
MB	0.19	0.09	0.26
MORTGAGE	0.89	0.78	0.03
NT	0.006	0.007	0.007
PY	0.027	0.014	0.018
QUAKE	0.04	0.03	0.04
TREASURY	0.77	0.51	0.17
WANKARA	0.002	0.002	0.002
AVERAGE	6.87	6.76	6.60

for the regression datasets. For this critical parameter, no large deviations appeared in the results of the proposed method. This further enhances the robustness and reliability of the proposed technique.

Furthermore, in the

Table 6. Precision, recall, and f-score for a series of classification datasets.

RBF-KMEANS DATASET	IRBF-100 PRECISION	RECALL	F-SCORE	PRECISION	RECALL	F-SCORE
APPENDICITIS	0.80	0.77	0.76	0.79	0.74	0.78
AUSTRALIAN	0.67	0.61	0.58	0.79	0.76	0.76
BALANCE	0.74	0.76	0.64	0.75	0.78	0.76
BANDS	0.52	0.51	0.48	0.58	0.57	0.56
HEART	0.68	0.69	0.67	0.86	0.85	0.85
IONOSPHERE	0.84	0.81	0.81	0.92	0.89	0.90
LIVERDISORDER	0.65	0.64	0.64	0.72	0.71	0.71
MAMMOGRAPHIC	0.81	0.81	0.81	0.83	0.83	0.82
PARKINSONS	0.76	0.68	0.69	0.85	0.80	0.81
PIMA	0.72	0.67	0.68	0.75	0.70	0.71
SAHEART	0.65	0.61	0.61	0.70	0.66	0.67
SEGMENT	0.43	0.39	0.39	0.58	0.53	0.53
SPIRAL	0.56	0.56	0.55	0.70	0.70	0.70
WDBC	0.93	0.91	0.92	0.96	0.94	0.95
WINE	0.74	0.65	0.66	0.93	0.93	0.92
Z_F_S	0.85	0.84	0.83	0.96	0.97	0.96
ZO_NF_S	0.90	0.90	0.90	0.95	0.95	0.95

, the metrics of precision, recall, and f-score are shown for a series of classification datasets and for the proposed method (IRBF-100) and the classic method for training RBF networks (RBF-KMEANS). In these experimental results, the reader can see the superiority of the proposed technique over the traditional method of training RBF networks.

4. Conclusions

In the present work, a two-stage hybrid method was proposed to efficiently identify the parameters of RBF neural networks. In the first stage of the method, a technique rooted in particle swarm optimization was used to efficiently identify a reliable interval of values for the neural network parameters. In the second stage of the method, an intelligent global optimization technique was used to locate the neural network parameters within the optimal value interval of the first stage. In this work, a genetic algorithm was used in the second phase, but any global optimization method could be used in its place.

The method was applied to a multitude of classification and regression problems from the relevant literature. In almost all cases, the proposed method significantly outperformed the other machine learning models, and on average, the improvement in the error on the test sets was of the order of 40% relative to the established RBF training method. Moreover, the method is quite robust with respect to the basic parameters since any changes in the parameter values do not significantly affect its performance. Furthermore, the method can efficiently locate the value interval of network parameters without any prior knowledge about the type of training data or whether it is a classification or a regression problem. However, the proposed technique is significantly more time-consuming than the traditional training technique, as it requires computational time for both of its phases. However, this effect can be overcome to some extent by the use of modern parallel computing techniques.

The method could be extended by the use of other techniques of training the parameters in RBF networks, such as, for example, the differential evolutionary method [107]. Furthermore, more efficient methods of terminating the first stage of the method could be used, as finding a suitable interval of values for the network parameters requires many numerical calculations.