A Two-Phase Evolutionary Method to Train RBF Networks

Abstract

This article proposes a two-phase hybrid method to train RBF neural networks for classification and regression problems. During the first phase, a range for the critical parameters of the RBF network is estimated and in the second phase a genetic algorithm is incorporated to locate the best RBF neural network for the underlying problem. The method is compared against other training methods of RBF neural networks on a wide series of classification and regression problems from the relevant literature and the results are reported.

1. Introduction

In machine learning, many practical problems appear such as classification and regression problems. A good programming tool that can be used to tackle this problem is radial basis function (RBF) networks [1]. These networks typically are expressed as a function:

$$y\left(x\right)=\sum _{i=1}^k{w}_i\varphi \left(∥x-c_i∥\right)$$

(1)

where $\overrightarrow{x}$ is the input pattern, the vector $\overrightarrow{w}$ is called the weight vector and $y\left(x\right)$ is the predicted value of the network. RBF networks are feedforward neural networks [2] with three computational layers:

• The input layer, where the problem is presented in the form of patterns to the neural network.
• The processing layer, where a computation is performed using the Gaussian processing units $\varphi \left(x\right)$. These units can have many forms in the relevant literature but the most used form is the Gaussian function expressed as:

  $$\varphi \left(x\right)=exp\left(-\frac{{\left(x-c\right)}^2}{{\sigma }^2}\right)$$

  (2)
  The value $\varphi \left(x\right)$ depends only on the distance of vector $\overrightarrow{x}$ from some other vector $\overrightarrow{c}$, which typically is called centroid.
• The output layer where the output of every function $\varphi \left(x\right)$ is multiplied by a corresponding weight value $w_i$.

RBF networks have been used in many classification and regression problems from the areas of physics [3,4,5,6], medicine [7,8,9], solution of differential equations [10,11], chemistry [12,13,14], economics [15,16,17], digital communications [18,19], etc. Furthermore, recently, RBF networks have been used in more difficult problems such as the authentication assurance of meat products [20], trajectory tracking for electrohydraulic servo systems, identification of geographical origin for foods [21], prediction of solution gas–oil ratio of crude oil systems [22], prediction of occurrences of haloketones in tap water [23], health monitoring [24], etc. Because of the extensive use of RBF networks, many methods have been proposed in the recent literature to enhance them. There are methods that parallelize the RBF networks [25,26], methods that improve the initialization of the RBF parameters [27,28,29], methods that alter the architecture of the network [30,31,32], methods aimed to locate the best set of RBF parameters with global optimization techniques [33,34,35], etc. This article transforms the problem of RBF training into an optimization problem and applies a modified genetic algorithm technique to solve it. The global optimization problem is defined as:

$$min\left(E\left(y\right)\right)=\sum _{i=1}^m{\left(y\left(x_i\right)-t_i\right)}^2$$

(3)

where m is the total number of input patterns and $t_i$ is the output for pattern $x_i$. The suggested approach has two phases: firstly, reasonable bounds for the RBF parameters are estimated using the k-means [36] algorithm and in the second phase, the modified algorithm is used to solve the problem of Equation (3) inside the bounds located in the first phase.

The rest of this paper is organized as follows: in Section 2 the proposed method is described, in Section 3 the conducted experiments are listed and the proposed method is compared against the traditional training of RBF networks and finally, in Section 4 some conclusions are derived.

2. Method Description

The proposed method can be divided into two main phases: during the first phase, an approximation for the bound of RBF parameters is made using the k-means algorithm; in the second phase, the optimization problem is solved using a modified genetic algorithm. These phases are outlined in detail in the following subsections.

2.1. Bound Location Phase

The proposed genetic algorithm has chromosomes with dimension $(d+1)\times k$, where d is the dimension of the input problem, i.e., the dimension of the vector $\overrightarrow{x_i}$ in Equation (3), and k is the total number of processing units of the RBF network. The layout of each chromosome is presented in

Table 1. The layout of the chromosomes in the proposed genetic algorithm.

${\mathit{c}}_{11}$

${\mathit{c}}_{12}$

…

${\mathit{c}}_{1\mathit{d}}$

${\mathit{\sigma }}_1$

${\mathit{c}}_{21}$

${\mathit{c}}_{22}$

…

${\mathit{c}}_{2\mathit{d}}$

${\mathit{\sigma }}_2$

…

${\mathit{c}}_{\mathit{k}1}$

${\mathit{c}}_{\mathit{k}2}$

…

${\mathit{c}}_{\mathit{k}\mathit{d}}$

${\mathit{\sigma }}_{\mathit{k}}$

. Every center $\overrightarrow{c_i}$ in Equation (1) is a vector of dimension d and an additional parameter is also reserved for the parameter $\sigma$ of every $\varphi \left(x\right)$ function. The centroids and the corresponding variances are estimated using the k-means algorithm that is described in Algorithm 1. The value ${\sigma }_i$ for every ${\varphi }_i\left(x\right)$ is calculated as:

$${\sigma }_i=\sum _{j=1}^d{s}_{ij}^2$$

(4)

After the estimation of $c_i$ and ${\sigma }_i$, the vectors $\overrightarrow{L},\overrightarrow{R}$ with dimension $(d+1)\times k$ are constructed. These vectors will serve as the bounds for the chromosomes of the genetic population. These vectors are constructed using the following procedure:

• Setm = 0
• Set$F>1$
• For$i=1..k$do

  (a)

  For$j=1..d$do

  i.

  Set$L_m$ = $-F\times c_{ij}$, $R_m$ = $F\times c_{ij}$

  ii.

  Set$m=m+1$

  (b)

  EndFor

  (c)

  Set$L_m=-F\times {\sigma }_i$, $R_m=F\times {\sigma }_i$

  (d)

  Set$m=m+1$

• EndFor

Algorithm 1: The k-means algorithm.

Repeat (a) $S_j=\left\{\right\},j=1..k$ (b) For every sample $x_i$ Do Set$j^{\ast }={min}_{i=1}^k\left\{D\left(x_i,c_j\right)\right\}$, where $j^{\ast }$ is the nearest center for sample $x_i$. Set$S_{j^{\ast }}=S_{j^{\ast }}\cup \left\{x_i\right\}$. (c) EndFor (d) For every center $c_j$ Do Set$M_j$ = number of elements in $S_j$ Update$c_j$ $$c_j=\frac{1}{M_j}\sum _{i=1}^{M_j}{x}_i$$ (e) EndFor Calculate the corresponding variances $$s_j^2=\frac{{\sum }_{i=1}^{M_j}{\left(x_i-c_j\right)}^2}{M_j}$$ Terminate when $c_j$ no longer changes.

2.2. Main Algorithm

The genetic algorithm used here is based on the algorithm denoted as GA$\left(c_{r1},l\right)$ in the Kaelo and Ali’s paper [37], with a modified stopping rule as proposed in [38]. The basic steps of the main algorithm are given below:

• Initialization Step

  (a)

  Read the train set with m patterns of d dimension.

  (b)

  Setk the number of nodes for the RBF network.

  (c)

  Estimate the vectors $\overrightarrow{L},\overrightarrow{R}$ using the procedure of Section 2.1.

  (d)

  Initialize a genetic population of $N_C$ random chromosomes inside $[L,R]$.

  (e)

  Set the selection rate $p_s\in [0,1]$, the mutation rate $p_M\in [0,1]$, $iter=0$ and $i_{max}$ the maximum number of generations.

• Evaluation Step
  For every chromosome g calculate the fitness $f_g$ using the procedure defined in Section 2.3.
• Genetic operations step
  During this step three genetic operations are performed: selection, crossover and mutation.

  (a)

  Selection procedure. Firstly, the chromosomes are sorted with relevance to their corresponding fitness value. The best $\left(1-p_s\right)\times N_c$ are transferred without change to the next generation and the remaining ones are substituted by offspring created through the crossover procedure. In the crossover procedure, the mating parent are selected using a tournament selection for every parent. The tournament selection is as follows:

  • Select a set of $T>2$ chromosomes from the population.
  • Return the chromosome with the best fitness value in that subset.

  (b)

  Crossover procedure. For every pair $(z,w)$ of selected parents create two new offspring $\tilde{z}$ and $\tilde{w}$:

  $$\begin{array}{ccc}\hfill \widetilde{z_i}& =& a_i{z}_i+\left(1-a_i\right)w_i\hfill \\ \hfill \widetilde{w_i}& =& a_i{w}_i+\left(1-a_i\right)z_i\hfill \end{array}$$

  (5)

  with $a_i$ a random number and $a_i\in [-0.5,1.5]$ [37]. This crossover scheme will be able to better explore the search space of the train error.

  (c)

  Mutation procedure. For every element of each chromosome, create a random number $r\in \left[0,1\right]$. If $r\le p_m$, then change that element randomly. The mutation is performed in a way similar to other approaches of genetic algorithms [38] and it is described in Section 2.5.

  (d)

  Replace the $p_s\times N_c$ worst chromosomes in the population with the generated offsprings.

• Termination Check Step

  (a)

  Set$iter=iter+1$

  (b)

  Terminate if the termination criteria of Section 2.4 are satisfied, else Goto Evaluation Step.

2.3. Fitness Evaluation

In this step, a valid RBF network $y\left(x\right)={\sum }_{i=1}^k{w}_i\varphi \left(∥x-c_i∥\right)$, is created using the chromosome g and is subsequently trained using the typical training procedure for RBF networks. The main steps to calculate the fitness $f_g$ of a chromosome g are the following:

• Decode the chromosome g to the parts (centers and variances) of the RBF network as defined by the layout of Table 1.
• Calculate the output vectors $w_1,w_w,\dots ,w_k$ by solving the induced system of equations:

  (a)

  Set$W=w_{kj}$ the matrix of k weights, $\mathsf{\Phi }={\varphi }_j\left(x_i\right)$ and $T=\left\{t_i\right\}$.

  (b)

  Solve:

  $${\mathsf{\Phi }}^T\left(T-\mathsf{\Phi }{W}^T\right)=0$$

  (6)

  $$W^T={\left({\mathsf{\Phi }}^T\mathsf{\Phi }\right)}^{-1}{\mathsf{\Phi }}^{T}T={\mathsf{\Phi }}^{†}T$$

  (7)

  The matrix ${\mathsf{\Phi }}^{†}={\left({\mathsf{\Phi }}^T\mathsf{\Phi }\right)}^{-1}{\mathsf{\Phi }}^T$ is the pseudo-inverse of $\mathsf{\Phi }$, with

  $${\mathsf{\Phi }}^{†}\mathsf{\Phi }=I$$

  (8)

• Set$f_g=$${\sum }_{i=}^m{\left(y\left(x_i\right)-t_i\right)}^2$

2.4. Stopping Rule

Define as $g_{best}$ the best chromosome in the population and define as ${\sigma }^{\left(\mathrm{iter}\right)}$ the variance of best fitness $f\left(g_{\mathrm{best}}\right)$ at generation iter. If fitness $f\left(g_{\mathrm{best}}\right)$ has not improved for a number of generations, then probably the algorithm should terminate. Hence, the termination rule is defined as:

$$\mathrm{iter}\ge i_{\max }\mathrm{OR}{\sigma }^{\left(\mathrm{iter}\right)}\le \frac{{\sigma }^{\left(\mathrm{klast}\right)}}{2}$$

(9)

where $\mathrm{klast}$ is the last generation where a new minimum was found.

2.5. Mutation Procedure

Let $w=\left(w_1,w_2,\dots ,w_n\right)$ be the chromosome to be mutated. The proposed mutation procedure modifies $w_i$ with probability $p_m$ and the resulting element $w_i^{\prime }$ is given by

$$w_i^{\prime }=\left\{\begin{array}{cc}{w}_i+\mathsf{\Delta }\left(\mathrm{iter},R_i-w_i\right),\mathrm{if}& t>\frac{1}{2}0\\ w_i-\mathsf{\Delta }\left(\mathrm{iter},w_i-L_i\right),\mathrm{otherwise}\end{array}\right.$$

(10)

where t is a random number with $t\in [0,1]$. The function $\mathsf{\Delta }(\mathrm{iter},y)$ is given by:

$$\mathsf{\Delta }(\mathrm{iter},y)=y\left(1-r^{\left(1-\frac{\mathrm{iter}}{\mathrm{ITERMAX}}\right)b}\right)$$

(11)

where r is a random number in $[0,1]$ and b controls the change of element $w_i$. In the proposed algorithm the value $b=5$ was used.

3. Experiments

In order to evaluate the performance of the proposed method, comparative experiments were performed on a series of well-known classification and regression datasets from the relevant literature.

3.1. Experimental Setup

The RBF network was coded in ANSI C++, using the Armadillo library [39] and the optimization was performed using the genetic optimization method of the optimization package OPTIMUS, that is freely available from https://github.com/itsoulos/OPTIMUS/ (accessed on 18 January 2021). Furthermore, to have more reliability in the results the commonly used method of 10-fold cross-validation was used, which means that the original data were randomly partitioned into 10 equally sized subsamples. Subsequently, 10 independent experiments were conducted: in each experiment one subsample was used as the testing data and all the others as the training data. The average error on the test data was the total test error. All the experiments were executed 30 times with different initialization for the random generator each time. The random generator used was the function drand48() of the C programming language. The execution environment was an Intel Xeon E5-2630 multicore machine using the OpenMP library [40] for parallelization and the Ubuntu Linux operating system. The parameters for the genetic algorithm are displayed in

Table 2. Experimental parameters.

Parameter	Value
k	10
$N_c$	200
$p_s$	0.90
$p_m$	0.05
F	3.0
$i_{max}$	200

. The parameters of the method were chosen so that there is a balance between speed and efficiency of the method.

3.2. Experimental Datasets

The classification problems used for the experiments were found in most cases on two Internet databases:

• UCI dataset repository, https://archive.ics.uci.edu/ml/index.php (accessed on 18 January 2021)
• Keel repository, https://sci2s.ugr.es/keel/datasets.php (accessed on 18 January 2021) [41].

The following classification datasets were used:

• The Alcohol dataset, which is related to alcohol consumption [42].
• The Appendicitis dataset, proposed in [43].
• The Australian dataset [44], which refers to credit card applications.
• The Balance dataset [45], which is used to predict psychological states.
• The Cleveland dataset, used in various papers to detect heart disease [46,47].
• The Dermatology dataset [48], which is used for differential diagnosis of erythematosquamous diseases.
• The Glass dataset, which contains glass component analysis and has been used in a variety of papers [49,50].
• The Hayes roth dataset [51].
• The Heart dataset [52], used to detect heart disease.
• The HouseVotes dataset [53], which is about votes in the U.S. House of Representatives.
• The Ionosphere dataset, which contains data from the Johns Hopkins Ionosphere database and has been studied in a number of papers [54,55].
• The Liverdisorder dataset [56], used to detect liver disorders in people using blood analysis.
• The Mammographic dataset [57], used to identify the severity of a mammographic mass lesions.
• The Parkinson dataset, which is composed of a range of biomedical voice measurements from 31 people, 23 with Parkinson’s disease (PD) [58].
• The Pima dataset [59], used to detect the presence of diabetes.
• The Popfailures dataset [60], related to climate model simulation crashes.
• The Regions2 dataset, which was created from liver biopsy images of patients with hepatitis C [61].
• The Ring dataset [62], a 20-dimension problem with two classes, where each class is drawn from a multivariate normal distribution.
• The Saheart dataset [63], used to detect heart disease.
• The Segment dataset [64], which contains patterns from a database of seven outdoor images (classes).
• The Sonar dataset [65]. The task is to discriminate between sonar signals bounced off a metal cylinder.
• The Spiral dataset, which is an artificial dataset containing 1000 two-dimensional examples that belong to two classes.
• The Tae dataset [66], which concerns evaluations of teaching performance.
• The Thyroid dataset, which concerns thyroid disease records [67].
• The Wdbc dataset [68], which contains data from breast tumors.
• The Wine dataset, used to detect through chemical analysis the origin of wines in various research papers [69,70].
• The EEG dataset from [71], which consists of five sets (denoted as Z, O, N, F and S), each containing 100 single-channel EEG segments each lasting 23.6 s. With different combinations of these sets the produced datasets are Z_F_S, ZO_NF_S, ZONF_S.
• The Zoo dataset [72], where the task is to classify animals in seven predefined classes.

The regression datasets are in most cases available from the StatLib URL ftp://lib.stat.cmu.edu/datasets/index.html (accessed on 18 January 2021):

• The Abalone dataset [73], which can be used to obtain a model to predict the age of abalone from physical measurements.
• The Airfoil dataset, which is used by NASA for a series of aerodynamic and acoustic tests [74].
• The Anacalt dataset [75], which contains information about the decisions taken by a supreme court.
• The BK dataset [76], used to estimate the points in a basketball game.
• The BL dataset, which can be downloaded from StatLib and contains data from an experiment on the effects of machine adjustments on the time it takes to count bolts.
• The Concrete dataset, used to measure the concrete compressive strength [77].
• The Housing dataset, taken from the StatLib library, which is maintained at Carnegie Mellon University, and described in [78].
• The Laser dataset, used in laser experiments. It has been obtained from the Santa Fe Time Series Competition Data repository.
• The MB dataset, available from Smoothing Methods in Statistics [77].
• The NT dataset, which contains data from [79] that examined whether the true mean body temperature is 98.6 F.
• The Quake dataset, whose objective is to approximate the strength of a earthquake. It has been obtained from the Bilkent University Function Approximation Repository.

3.3. Experimental Results

The results for the classification datasets are listed in

Table 3. Classification error for different datasets.

Dataset	KRBF	GRBF	Proposed
Alcohol	46.63%	52.30%	21.86%
Appendicitis	12.23%	16.83%	16.03%
Australian	34.89%	41.79%	22.97%
Balance	33.42%	38.02%	12.88%
Cleveland	67.10%	67.47%	51.75%
Dermatology	62.34%	61.46%	37.37%
Glass	50.16%	61.30%	49.16%
Hayes Roth	64.36%	63.46%	35.26%
Heart	31.20%	28.44%	17.80%
HouseVotes	6.13%	11.99%	3.67%
Ionosphere	16.22%	19.83%	10.33%
Liverdisorder	30.84%	36.97%	28.73%
Mammographic	21.38%	30.41%	17.25%
Parkinsons	17.42%	33.81%	17.37%
Pima	25.78%	27.83%	24.00%
Popfailures	7.04%	7.08%	5.44%
Regions2	38.29%	39.98%	25.81%
Ring	21.65%	50.36%	2.09%
Saheart	32.19%	33.90%	29.38%
Segment	59.68%	54.25%	39.44%
Sonar	27.85%	34.20%	19.62%
Spiral	44.87%	50.02%	18.98%
Tae	60.07%	61.78%	52.44%
Thyroid	10.52%	8.53%	7.12%
Wdbc	7.27%	8.82%	5.29%
Wine	31.41%	31.47%	8.67%
Z_F_S	13.16%	23.37%	4.21%
ZO_NF_S	9.02%	22.18%	4.17%
ZONF_S	4.03%	17.41%	2.18%
ZOO	21.93%	33.50%	9.00%

and for the regression datasets the results are reported in

Table 4. Regression error for different datasets.

Dataset	KRBF	GRBF	Proposed
Abalone	2559.48	4161.66	1960.22
Airfoil	5.49	18.15	0.58
Anacalt	11.628	5.58	0.003
BK	0.17	0.21	0.23
BL	0.05	0.019	0.0009
Concrete	1.15	1.50	0.52
Housing	2884.09	4784.50	693.22
Laser	2.35	6.94	1.04
MB	11.33	2.44	0.63
NT	72.14	0.22	0.09
Quake	15.36	171.43	7.86

. For the first case, the average classification error is reported and for the case of regression datasets, the total test error is reported. The column KRBF denotes the classic RBF training method, GRBF denotes the method proposed in [80] and the column “proposed” denotes the proposed method. The KBF simply consists of two phases: in the first phase, the centers and variances are estimated through the k-means algorithm and in the second phase, a system of equations is solved to obtain the weights $w_i$ of the RBF network.

From the experimental results, it is clear that the proposed method is significantly superior to other methods in almost all datasets. In the proposed method, the appropriate initialization interval was found for the parameters of RBF using k-means. A parallel genetic algorithm was then applied to this previous value range, creating a variety of neural networks. This combination of techniques obviously has very good results as it combines a very efficient clustering method and an excellent optimization method that is ideally parallelized. Of course, the new method requires much more execution time, due to the presence of the genetic algorithm, but the parallel execution of the software drastically reduces this time. Furthermore, in order to study the effectiveness of the selection of parameter F an additional experiment was conducted, where the best fitness of the genetic algorithm is plotted for the Wine problem. The outcome of this experiment is graphically outlined in Figure 1. The graph shows that the behavior of the proposed method does not change significantly for different values of the parameter F. Furthermore, the plot for the Wine dataset of best, worst and average fitness for $F=3$ is shown in Figure 2. An additional experiment was performed to evaluate the effect of the parameter change k on the results. In Figure 3, the plot for different values of k for the Housing dataset is outlined and in Figure 4 the same experiment is shown for the Z_F_S classification dataset. Of course, from the value $k=4$ onwards, the error falls but not at the same rate. The proposed value $k=10$ was used in all datasets in order to have a balance between the speed and the efficiency of the method. Finally, the classification performance was evaluated based on two evaluation metrics: precision and recall for some datasets as shown in

Table 5. Comparison of precision and recall between the traditional RBF and the proposed method for some datasets.

Dataset	Precision KRBF	Recall KRBF	Precision Proposed	Recall Proposed
Housevotes	90.61%	95.60%	96.44%	94.25%
Spiral	55.53%	55.93%	82.93%	84.66%
ZONF_S	92.76%	87.23%	97.27%	94.55%

. In these results, the precision of the proposed method showed the best classification performance under different datasets. We found the same trends in recall metric in two of three datasets (spiral and EEG datasets).

4. Conclusions

A two-phase method was proposed in this article to train RBF neural networks for classification and regression problems. Firstly, a commonly used clustering method was used to estimate an interval for the critical parameters of the neural network. Subsequently, a parallel genetic algorithm was incorporated to locate the best RBF network with good generalization capabilities. The used algorithm was coded using ANSI C++ and open source libraries such as the Armadillo library and the OpenMP library for parallelization. Future research may include:

• Using parallel methods for the k-means clustering phase of the method.
• Dynamic selection of k in k-means algorithm.
• More advanced stopping rules for the genetic algorithm.
• Replacing the genetic algorithm with other optimization methods such as particle swarm optimization, ant colony optimization, etc.