Optimization by Context Refinement for Development of Incremental Granular Models

Abstract

Optimization by refinement of linguistic contexts produced from an output variable in the construction of an incremental granular model (IGM) is presented herein. In contrast to the conventional learning method using the backpropagation algorithm, we use a novel method to learn both the cluster centers of Gaussian fuzzy sets representing the symmetry in the premise part and the contexts of the consequent part in the if–then fuzzy rules. Hence, we use the fundamental concept of context-based fuzzy clustering and design with an integration of linear regression (LR) and granular fuzzy models (GFMs). This GFM is constructed based on the association between the triangular membership function produced both in the input–output variables. The context can be established by the system user or using an optimization method. Hence, we can obtain superior performances based on the combination of simple linear regression and local GFMs optimized by context refinement. Experimental results pertaining to coagulant dosing in a water purification plant and automobile miles per gallon prediction revealed that the presented method performed better than linear regression, multilinear perceptron, radial basis function networks, linguistic model, and the IGM.

1. Introduction

Owing to the rapid growth of various application problems, several studies have been conducted on fuzzy models. It is now acknowledged that real-world problems require intelligent methods that integrate the methodology of different models. To solve various application challenges, it is important to design complementary hybrid intelligent systems by integrating multiple computing technologies in a synergistic rather than exclusive manner. A representative method is neuro-fuzzy inference modeling [1,2]. Neural networks adapt to changing environments, whereas fuzzy models deal with fuzzy inference and knowledge-based decision making.

The basic concept of fuzzy clustering has been used in several studies to focus on fuzzy modeling, resulting in the concept of granular fuzzy models (GFMs) [3,4,5,6,7,8]. These models is based on information granules. These information granules (IG) comprise linguistic information produced from both input–output variables. The IG in the design of GFMs are obtained using specialized context-based fuzzy c-means clustering. Unlike context-free clustering, the fuzzy c-means clustering without context produces clusters. When the contexts are generated, then clustering occurs within a specified context. A system modeling method using a context-based fuzzy clustering algorithm has been successfully applied. Yeom [6] proposed a performance evaluation method based on the concept of coverage and specificity in developing a GFM. The performance is typically measured by the root mean square error (RMSE). Byeon [7] designed a Takagi–Sugeno–Kang (TSK) linguistic model. In that study, an intelligent predictor was designed to predict the direction of dinghy booms. Park [8] applied a linguistic model to water quality prediction. The main characteristics of the presented model are that it is user-centric and dwells upon the collection of user-oriented entities. Several studies have been conducted using the GFM [9,10,11,12,13,14,15,16,17,18].

In addition to several studies related to information granules [19,20,21,22,23,24], an incremental granular model (IGM) based on an integration of linear regression and a linguistic model has been proposed by Kwak. First, after performing a linear regression (LR), the error caused by the LR was predicted by the local granular model. Yeom [21] proposed an improved IGM based on granular networks with the compensation of error. Li [22] applied a patch-type sensor model based on a radial basis function network (RBFN) with incremental concept to estimate energy expenditure. Lee [23] presented a design method for a RBFN with incremental concept by linear regression and local RBFN for predicting the heating and cooling load in residual buildings problem. Kwak [24] proposed an IGM designed by context-based fuzzy c-means clustering with interval type-2 concept. This model possesses unique characteristics that preserve the homogeneity and includes uncertainties associated with the fuzzification factor. Several studies have been conducted using the IGM [25,26,27].

Although the IGM has been applied successfully in several studies, a method to adjust the premise and consequent parts using the learning method in the IGM has not been considered. Thus, the purpose of this paper is to perform the systematic design of an IGM based on a hybrid learning method. The design process comprises several important steps involving the adjustment of linguistic contexts in the design of local GFMs, estimation of cluster centers of Gaussian fuzzy sets representing the symmetry in context-based fuzzy clustering, and optimization of premise and consequent parameters. Here, the contexts are adjusted by a derivative-based optimization method. In contrast to the conventional derivative-based optimization method, the advantages of the proposed method is that the only linguistic contexts with weight information are adjusted by gradient descent method. After adjusting the contexts, the premise parameters are automatically updated by context-based fuzzy clustering. This method has unique characteristics in comparison to the conventional method.

The remainder of this paper is organized as follows. Section 2 presents the IGM constituents, including linear regression, context-based fuzzy c-means clustering, and a local GFM. Furthermore, this section includes the architecture and design procedure of the IGM. Section 3 provides the refinement of linguistic contexts in the development of the IGM. In Section 4, experiments performed on a coagulant dosing process in a water purification plant and automobile miles per gallon (mpg) prediction are described [28,29]. Finally, conclusions are provided in Section 5.

2. IGM

The components of the IGM include linear regression as the global part, a GFM as the local part, and context-based fuzzy c-means clustering for generating if–then rules. This section highlights the tendency of system modeling in proceeding to the simplest possible model, evaluating its performance, and then completing a series of necessary improvements.

2.1. Linear Regression

The experimental data under discussion are assumed to comprise 2D input–output data pairs. These data pairs are provided in the following format: $\{x_k,y_k\},k=1,2,\cdots ,N.$ Here, $x_k$ is the k-th input vector and $y_k$ is the k-th output. Linear regression is expressed in the following standard form:

$$z_k=a^{\mathrm{T}}{x}_k+a_0,$$

(1)

where the regression coefficients are expressed by $a$ and $a_0$. The coefficients are obtained using the standard least-squares error method. Here, $a^{\mathrm{T}}$ consists of two coefficients [a₁, a₂]^T. The improvement of the model occurs in which the local part is based on the input-error data $\{x_k,e_k\}$, where the error is $e_k=y_k-z_k$. Subsequently, the if–then fuzzy rules and contexts of triangular membership functions are obtained by the context-based fuzzy clustering approach.

2.2. Context-Based Fuzzy C-Means Clustering

The context-based fuzzy clustering algorithm is an approach for estimating the clusters to maintain the association characteristics [30,31,32]. The following reiterates the nature of context-based fuzzy clustering. This clustering is realized for each context ${\mathrm{W}}_1{,\mathrm{W}}_2,\cdots {,\mathrm{W}}_{\mathrm{p}}$ through the statistical distribution of the output space. The contexts in the traditional GFM are created by a set of triangular fuzzy sets spaced equally along the output. It can be difficult to estimate the clusters because some contexts are associated with data scarcity.

It can be difficult to produce if–then fuzzy rules in context-based fuzzy clustering approach. To create flexible contexts, we used a characteristics of probabilistic distribution for the output in shown in Figure 1. Figure 1 shows the generation of linguistic contexts obtained by probability density function (b) and conditional density function (c) based on the statistical distribution of error (a) obtained by linear regression. Finally, contexts with a triangular fuzzy set were generated.

As shown in Figure 1d, the generated contexts are represented as a linguistic label of error.

The partitioning matrix obtained by the t-th context is introduced as follows:

$${\mathrm{U}(\mathrm{W}}_{\mathrm{t}})=\left\{u_{ik}\in \left[0,1\right]|{\displaystyle {\displaystyle \sum }_{i=1}^c{u}_{ik}=w_{tk} \forall k}\right\},$$

(2)

where $w_{tk}$ is the membership grade of the k-th data point in the t-th linguistic context; $u_{ik}$ is the grades corresponding to the k-th data point in the i-th cluster center. The loss function can be defined as

$$J={\displaystyle {\displaystyle \sum }_{i=1}^c}{\displaystyle {\displaystyle \sum }_{k=1}^N{u}_{ik}^m}||x_k-v_i|{|}^2,$$

(3)

where $v_i$ is the cluster’s center of the i-th cluster. $\Vert x_k-v_i\Vert$ is computed by distance between the k-th input and the i-th center. Here, the fuzzification factor m is 2. The objective function is accomplished by adjusting the partition matrix and the cluster center’s membership grades. The update of the membership matrix is computed as [30,31,32]:

$$u_{tik}=\frac{w_t{}_k}{{\displaystyle {\displaystyle \sum }_{j=1}^c{\left(\frac{\Vert x_k-v_i\Vert }{\Vert x_k-v_j\Vert }\right)}^{\frac{2}{m-1}}}} i=1,2,\dots ,c, k=1,2,\dots ,N,$$

(4)

where $u_{tik}$ denotes the i-th center and the k-th data point corresponding the t-th linguistic context. The cluster center $v_i$ is computed as follows:

$$v_i=\frac{{\displaystyle {\displaystyle \sum }_{k=1}^N{u}_{tik}^m{x}_k}}{{\displaystyle {\displaystyle \sum }_{k=1}^N{u}_{tik}^m}}$$

(5)

Figure 2 shows the estimation of cluster centers corresponding to six contexts. After generating the contexts as shown in Figure 1, the cluster centers are obtained by context-based fuzzy c-means clustering. As shown in Figure 2, the data points (dot mark) included in each context and cluster centers (square mark) are distributed.

2.3. Local GFM

Figure 3 shows the architecture of the local GFM and the association between the cluster center and context. In Figure 3, the circle nodes represents the computation of membership degrees obtained in Equation (4). The GFM’s output is obtained as a triangular fuzzy membership function representing a fuzzy set. In other words, the GFM output $\hat{E}$ is fully represented by three parameters, $\hat{E}=\langle {\hat{e}}_{-},\hat{e},{\hat{e}}_{+}\rangle$. These parameters denotes lower bound, modal value, and upper bound, respectively.

Assuming a triangular fuzzy membership function for the contexts, the GFM’s output is expressed as [4,8]

$$\hat{E}{=\mathrm{W}}_1\otimes {\mathsf{\xi }}_1\oplus {\mathrm{W}}_2\otimes {\mathsf{\xi }}_2\oplus \dots \oplus {\mathrm{W}}_{\mathrm{p}}\otimes {\mathsf{\xi }}_{\mathrm{p}},$$

(6)

Equation (6) marked by $\otimes$ and $\oplus$ are used to compute that the basic calculations are performed by fuzzy numbers. ${\mathsf{\xi }}_k$ is the summation value of the activation values generated in the k-th linguistic context. Here, activation level denotes the summation of membership degrees corresponding to each context. That is, the three points of triangular fuzzy sets representing contexts and the activation values are multiplied, respectively. Thus, the output includes the lower value, modal value, and upper value with interval prediction.

The activation values were computed using Equation (4). The numeric bias term was used to eliminate possible systematic errors as follows:

$${\mathrm{W}}_0=\frac{1}{N}{\displaystyle {\displaystyle \sum }_{k=1}^N(e_k-{\hat{e}}_k)},$$

(7)

Here, the modeling errors $e_k$ obtained from LR are obtained by a global model. This value is calculated simply to eliminate errors. Consequently, the model output $\hat{E}(x_k)$ allows the characterization for three parameters:

$$\langle {\displaystyle {\displaystyle \sum }_{t=1}^p{\mathsf{\xi }}_t{\mathrm{W}}_{t-}+{\mathrm{W}}_0}, {\displaystyle {\displaystyle \sum }_{t=1}^p{\mathsf{\xi }}_t{\mathrm{W}}_t+{\mathrm{W}}_0}, {\displaystyle {\displaystyle \sum }_{t=1}^p{\mathsf{\xi }}_t{\mathrm{W}}_{t+}+{\mathrm{W}}_0}\rangle$$

(8)

2.4. IGM

The flow chart of the IGM design is visualized in Figure 4. Adopting a linear regression structure, we compensated the modeling errors through local rules of the GFM that represent the local nonlinearity of the model to be considered [9,11,28]. Figure 5 shows the architecture of the IGM.

The construction procedures of the IGM shown in Figure 5 are as follows:

[Step 1]

Use linear regression on the numerical data points. Subsequently, the errors $e_k=y_k-z_k$ are obtained by the difference between the desired and linear regression outputs.

[Step 2]

Obtain the input and error pairs $\{x_k,e_k\}$. These error values are employed as the output data in the use of the local GFM. Subsequently, the contexts in the error space are generated. The linguistic contexts are produced as shown in Figure 1.

[Step 3]

Estimate the clusters using context-based fuzzy clustering approach.

[Step 4]

Calculate the aggregation values by the linear summation of the activation levels and the context weight. Consequently, the model output results in fuzzy number with a triangular type.

[Step 5]

Integrate the linear regression output and granular results of the GFM. Hence, the prediction result is expressed as $\hat{Y}=z\oplus \hat{E}$

3. Refinement of Contexts in IGM Design

We present the optimization of the context refinement in this Section. This optimization method comprises two steps, as shown in Figure 6. First, we used the gradient descent method to adjust the contexts from the error between the desired and GFM outputs. Next, we obtained new cluster centers using context-based fuzzy clustering approach. The number of iteration to operate in the loop is 20.

We can consider the optimization methods where the minimization is performed by the centers of the triangular membership function as the context. The maximization of the average match between the granular output of the GFM and the available numerical desired output is straightforward. We assumed that Y is an output of the GFM for $x_k$, $\mathrm{Y}\left(x_k\right)$.

As described by the triangular fuzzy sets, we calculated the degrees as follows:

$$\max \frac{1}{N}{{\displaystyle \sum }}_{k=1}^{N}Y\left(x_k\right)\left({\mathrm{target}}_k\right)$$

(9)

Furthermore, we can consider minimizing the average values of the output for the corresponding inputs as follows:

$$\min \frac{1}{N}{{\displaystyle \sum }}_{k=1}^N\left({\mathrm{b}}_k-a_k\right),$$

(10)

where $a_k$ and ${\mathrm{b}}_k$ are the values with lower and upper bounds of the triangular membership function produced for $x_k$, respectively. We performed the optimization by assuming that the successive contexts can overlap at the half-point level. Therefore, the optimization focuses on the median values of the triangular fuzzy membership function of the context. When these modal values have been updated, this clustering is performed. Subsequently, the iteration loop of the optimization is repeated. In gradient descent methods, the next point is adjusted as follows:

$$w\left(new\right)=w\left(old\right)+\mathsf{\Delta }w$$

(11)

The chain rule to obtain the gradient vector is applied as follows:

$$\mathsf{\Delta }{w}_i=-\eta ·\left(\frac{\partial E_k}{\partial w_i}\right)=-\eta ·\frac{\partial E_k}{\partial {\hat{y}}_k}·\frac{\partial {\hat{y}}_k}{\partial w_i}$$

(12)

$$\frac{\partial E_k}{\partial {\hat{y}}_k}=2\left({\hat{y}}_k-y_k\right),\frac{\partial {\hat{y}}_k}{\partial w_i}=u_i$$

(13)

Hence, the update formula is defined as follows:

$$\mathsf{\Delta }{w}_i=-2·\eta ·\left({\hat{y}}_k-y_k\right)·u_i$$

(14)

Figure 7 shows a flowchart pertaining to context refinement by gradient descent and cluster adjustment via clustering.

4. Experimental Results

We performed experiments using the proposed IGM with two optimization steps for coagulant dosing in a water purification plant and automobile mpg prediction in these experiments. The experimental results were compared with those of previous studies.

4.1. Coagulant Dosing in Water Purification Plant

The test data were collected at a water purification plant, and the water purification capacity was 1,320,000 ton/day [28]. We used 346 consecutive samples. The input comprised the turbidity, temperature, pH, and alkalinity. The output was polyaluminum chloride (PAC). To evaluate the resulting model, the dataset was partitioned into training and test data sets, respectively. In this experiment, 173 training data pairs were used to construct the IGM using the optimization method, and the 173 test datasets were used to validate the presented IGM.

Table 1. Comparison results for root mean square error (RMSE) for PAC prediction.

		No. of Rule (*: No. of Node)	Trn_RMSE	Tst_RMSE
Linear regression		-	3.508	3.578
Multilayer perceptron		45 *	3.191	3.251
RBFN based on context-based fuzzy c-means clustering [20]		45 *	3.048	3.219
Linguistic model [4] (p = 8)	c = 8	64	2.427	2.800
IGM [13]	p = c = 8	64	1.790	2.009
The proposed model	p = c = 6	36	2.124	2.271
	p = c = 7	49	1.862	2.093
	p = c = 8	64	1.640	1.976
	p = c = 9	81	1.631	2.105

lists the performance comparison for the training (Trn_RMSE) and test (Tst_RMSE) data for the PAC prediction of the coagulant dosing process. The number of iteration loops was 20, and the learning rate was 0.001. Figure 8 shows the prediction performance for the training and test datasets, respectively. As shown in Figure 8, the experimental results showed good prediction and generalization capability. Figure 9 shows the error distribution of PAC obtained by linear regression. As shown in Figure 9, it can be seen that the error distribution is concentrated near zero. Figure 10 shows the cluster centers corresponding to each context when the number of contexts was 8. As listed in Table 1, the prediction performance revealed that the IGM with the new optimization method performed better than linear regression, multilayer perceptron, linguistic model, and the IGM. As listed in Table 1, the IGM without optimization method showed good performance in comparison to the previous works. Furthermore, the experimental results revealed that the IGM with optimization method enhanced the performance of IGM itself. When p = c = 8, we obtained the best prediction performance in trial and error method. We followed the selection method of the number of node and rules in the previous works [28].

4.2. Automobile MPG Prediction

The automobile MPG prediction data are available from the UCI repository [29]. The dataset consists of 392 data pairs. The input comprised the weight, acceleration, model year, cylinder number, displacement, and horsepower. The dataset was partitioned into training and test datasets. In this experiment, 196 training data pairs and the remaining test datasets were used for model evaluation. The number of iteration loops and the learning rates used were the same as those used previously. Figure 11 shows the performance for the training and test datasets. Figure 12 shows the distribution of error obtained by linear regression. Figure 13 shows the centers corresponding to each context, when the number of contexts was 8.

Table 2. Performance comparison.

		No. of Rule	Trn_RMSE	Tst_RMSE
Linguistic model [4]		36	2.788	3.337
IGM [13]		36	2.418	3.175
The proposed model	p = c = 6	36	2.274	3.041
	p = c = 7	49	2.206	2.984
	p = c = 8	64	2.138	2.862

lists the performance comparison. As listed in Table 2, the experimental results revealed that the proposed IGM performed better than the linguistic model and IGM. When p = c = 8, we obtained the best prediction performance in trial and error method. We followed the selection method of the number of node and rules in the previous works [28].

4.3. Boston Housing Data

Next, we shall use Boston housing data set that deals with the problem of real estate price prediction. This data are available from the UCI repository [29]. In this example, we used twelve input variables except for one binary attribute. The total data include 506 data pairs. We divided the data set into training and testing data sets of equal size. In this study, we used 253 training data pairs, while the remaining testing data sets were used for model evaluation.

Table 3. Comparison results for RMSE for Boston housing data set.

	No. of Rule	RMSE (Training)	RMSE (Testing)
Linguistic model [4]	36	4.790	4.878
IGM [13]	36	3.371	3.649
The proposed method(p = c = 7)	49	3.106	3.273

listed the comparison results for RMSE for Boston housing data set. When p = c = 8, we obtained the best prediction performance in trial and error method. The proposed method outperformed the linguistic model and the basic IGM.

4.4. Discussion

The experimental results revealed that the proposed optimization method showed good performance in comparison to linguistic model and IGM itself for three data sets. The proposed method can be summarized by the following strength and weakness:

Strengths:

-

The incremental granular model has high prediction performance by combining linear regression and local granular fuzzy model.

-

The local granular fuzzy model generates the automatic if-then rules using context-based fuzzy clustering method from numerical data set.

-

The incremental granular model can enhance the prediction performance by combining the derivative-based optimization and context-based fuzzy clustering.

-

In contrast to the conventional back-propagation method, after adjusting the contexts by steepest descent method, the cluster centers in the premise part are estimated by using context-based fuzzy clustering method.

Weaknesses:

-

The number of contexts is obtained by trial and error method.

-

The number of cluster center per context are obtained by trial and error method.

-

As the number of data points increase, the number of rules also increase

-

The specific context can include the small data points, when the distribution of context is uniform.

5. Conclusions

We developed an optimization method for an IGM. This method comprised two stages: context refinement and cluster adjustment. In contrast to the conventional gradient descent method, we performed a hybrid learning method using the gradient descent method and clustering. The experimental results clearly indicated that the proposed IGM with the optimization method presented better generalization capability compared with the conventional methods and IGM. Therefore, we conclude that the new optimization method for IGM design is effective. Future studies will involve the optimal determination of cluster per context and new performance measures based on the concept of coverage and specificity.