The Fractional Differential Polynomial Neural Network for Approximation of Functions

Abstract

In this work, we introduce a generalization of the differential polynomial neural network utilizing fractional calculus. Fractional calculus is taken in the sense of the Caputo differential operator. It approximates a multi-parametric function with particular polynomials characterizing its functional output as a generalization of input patterns. This method can be employed on data to describe modelling of complex systems. Furthermore, the total information is calculated by using the fractional Poisson process.

1. Introduction

The Polynomial Neural Network (PNN) algorithm is one of the most important methods for extracting knowledge from experimental data and to locate its best mathematical characterization. The proposed algorithm can be utilized to analyze complex data sets with the objective to conclude internal data relationships and to impose knowledge about these relationships in the form of mathematical formulations (polynomial regressions). One of the most common types of PNN is the Group Method of Data Handling (GMDH) polynomial neural network created in 1968 by Professor Ivakhnenko at the Institute of Cybernetics in Kyiv (Ukraine).

Based on GMDH, Zjavka developed a new type of neural network called Differential Polynomial Neural Network (D-PNN) [1,2,3,4]. It organizes and designs some special partial differential equations, performing a complex system model of dependent variables. It makes a sum of fractional polynomial formulas, determining partial mutual derivative alterations of input variable combinations. This kind of retreatment is based on learning generalized data connections. Furthermore, it offers dynamic system models a standard time-series prediction, as the character of relative data allow it to employ a wider range of input interval values than defined by the trained data. In addition, the advantages of differential equation solutions facilitate a major variety of model styles. The principle of this type is similar to the artificial neural network (ANN) construction [5,6].

Fractional calculus is a section of mathematical analysis that deals with considering real number powers or complex number powers of the differentiation and integration operators. The integrals are of convoluted form and exhibit power-law type kernels. It can be viewed as an experimenter for special functions and integral transforms [7,8,9,10,11,12]. It is well known that the physical interview of the fractional derivative is an open problem today. In [13], the author utilized fractional operators, in the sense of the Caputo differential operator, to define and study the stability of recurrent neural network (NN). In [14], Gardner employed a discrete fractional calculus to study Artificial Neural Network Augmentation. In [15], Almarashi used neural networks with a radial basis function method to solve a class of initial boundary values of fractional partial differential equations. Recently, Jalab et al., applied the neural network method for finding the numerical solution for some special fractional differential equations [16]. Zhou et al. propoced a fractional time-domain identification algorithm based on a genetic algorithm [17], while Chen et al. studied the synchronization problem for a class of fractional-order chaotic neural networks [18].

Here, our aim is to introduce a generalization of the differential polynomial neural network utilizing fractional calculus. The fractional calculus is assumed in the sense of the Caputo differential operator. It approximates a multi-parametric function with particular polynomials characterizing its functional output as a generalization of input patterns. This method can be employed on data to describe modelling of complex systems [19].

2. Preliminaries

This section concerns with some basic preliminaries and notations regarding the fractional calculus. One of the most considerably utilized instruments in the theory of fractional calculus is provided by the Caputo differential operator.

Definition 2.1 The fractional (arbitrary) order integral of the function $h$ of order $\beta >0$ is defined by:

(1)

When $a=0,$ we write $I_a^{\beta }h\left(t\right)=h\left(t\right)\text{*} {\chi }_{\beta }\left(t\right),$ where $\left(\text{*}\right)$ denoted the convolution product (see [7]), ${\chi }_{\beta }\left(t\right)=\frac{t^{\beta -1}}{\text{Γ}\left(\beta \right)},t>0$ and ${\chi }_{\beta }\left(t\right)=0,t\le 0$ and ${\chi }_{\beta }\to \wp \left(t\right)$ as $\beta \to 0$ where $\wp \left(t\right)$ is the delta function.

Definition 2.2 The Riemann-Liouville fractional derivative of the function $h$ of order $0\le \beta <1$ is defined by:

(2)

Remark 2.1 [7]

$$D^{\beta }{t}^{\mu }=\frac{\text{Γ}\left(\mu +1\right)}{\text{Γ}\left(\mu -\beta +1\right)}{t}^{\mu -\beta },\mu >-1; 0<\beta <1$$

(3)

and:

$$I^{\beta }{t}^{\mu }=\frac{\text{Γ}\left(\mu +1\right)}{\text{Γ}\left(\mu +\beta +1\right)}{t}^{\mu +\beta },\mu >-1; \beta >0.$$

(4)

The Leibniz rule is:

(5)

Definition 2.3. The Caputo fractional derivative fractional derivative of order β>0 is defined, for a smooth function f by:

(6)

The local fractional Taylor formula has been generalized by many authors [20,21,22]. This generalization admits the following formula:

(7)

where ${}^{c}D^{\beta }{}_x$ is the Caputo differential operator and:

$${}^{c}D^{n\beta }{}_x=\underset{n-times}{\underbrace{{}^{c}D^{\beta }{}_x{}^{c}D^{\beta }{}_x\mathrm{...}{}^{c}D^{\beta }{}_x}}$$

(8)

3. Results

3.1. Proposed Method

The fractional differential polynomial neural network (FD-PNN) is based on an equation of the form:

(9)

where $u:=f\left(x_1,x_2,\mathrm{...},x_n\right)$ is a function of all input variables, $a,b_i,c_{ij},d_{ijk}$ are the polynomial coefficients. Solutions of fractional differential equations can be expressed in term of the Mittag-Leffler function:

(10)

Recently, numerical routines for Mittag-Leffler functions have been developed, e.g., by Freed et al. [23], Gorenflo et al. [24] (with MATHEMATICA), Podlubny [25] (with MATLAB), Seybold and Hilfer [26].

We proceed to form sum derivative terms changing the fractional partial differential equation (9) by applying different math techniques, e.g. fractional wave series, [27]:

$$\begin{array}{ll}{y}_i^{\beta }\hfill & =\frac{{(a_0+a_1{x}_1+a_2{x}_2+\mathrm{...}+a_n{x}_n+a_{n+1}{x}_1{x}_2+\mathrm{...})}^{\frac{m+\beta }{n}}}{b_0+b_1{x}_1+\mathrm{...}}\hfill \\ & =\frac{{\partial }^{m\beta }f\left(x_1,x_2,\mathrm{...},x_n\right)}{\partial x_1^{\beta }\partial x_2^{\beta }\mathrm{...}\partial x_m^{\beta }},\hfill \end{array}$$

(11)

where $n$ refers to the combined degree of $n-$ input variable polynomial of numerator; while $m$ indicates to the combined degree of denominator $w_t$ – weights of terms and $y_i^{\beta }$ is the output neuron. Note that when $\beta \to 1,$ Equation (11) reduces to Equation (4) in [4]. The fractional polynomials of fractional power (11), determining relations of $n$ -input variables, appear summation derivative terms (neurons) of a fractional differential equation. The numerator of Equation (11) is a complete $n$ -variable polynomial, which recognizes a new partial function $u$ of Equation (9). The denominator of Equation (11) is a fractional derivative part, which implies a fractional partial change of some input variables combination. Equation (11) indicates a aingle output for fixed fractional power. Each layer of the FD-PNN contains blocks. These blocks stress fractional derivative neurons. For each fractional polynomial of fractional order formulates the fractional partial derivative depending on the change of some input variables. Each block implicates a unique fractional polynomial which forms its output access into the next hidden layer (Figure 1). For example of a system of the form : input layer, first hidden layer, second hidden layer and output layer; we may use y₁^1/4 to perform its output to the first layer; y₂^1/2 to execute its output to the second hidden layer and y₃^3/4 to carry out the last y of the system in the output layer.

Figure 1. GMDH-PNN.

Figure 1. GMDH-PNN.

Let there be a network with two inputs, formulating one functional output value $y^{\beta }$, then, for special values of $\beta$, the sum derivative terms is:

(12)

we realize that $y^{\beta }$ includes only one block of two neurons, terms of both fractional derivative variables $x_1$ and $x_2.$Table 1 shows approximation errors (y-axis) of the trained network, i.e. differences of the true and estimated function, to random input vectors with dependent variables.

Table 1. Approximation values of $f\left(x_1,x_2\right)=x_1+x_2$.

Data	Actual Value	Approximate Value	Absolute Error
(1,0)	1	$y^1=y^{3/4}=\frac{3}{4}$	0.25
		$y^{1/2}=y^{1/4}=\frac{3}{4}$	0.25
(0,1)	1	$y^1=y^{3/4}=\frac{3}{4}$	0.25
		$y^{1/2}=y^{1/4}=\frac{3}{4}$	0.25
(1,1)	1	$y^1=1.5$	0.5
		$y^{3/4}=1.3$	0.3
		$y^{1/2}=1.1$	0.1
		$y^{1/4}=0.99$	0.01
(1/2,1/2)	1	$y^1=1.66$	0.66
		$y^{3/4}=1.6$	0.6
		$y^{1/2}=1.57$	0.57
		$y^{1/4}=1.53$	0.53
		$y^{0.1}=1.4$	0.4

Table 1. Approximation values of $f\left(x_1,x_2\right)=x_1+x_2$.

Data	Actual Value	Approximate Value	Absolute Error
(1,0)	1	$y^1=y^{3/4}=\frac{3}{4}$	0.25
		$y^{1/2}=y^{1/4}=\frac{3}{4}$	0.25
(0,1)	1	$y^1=y^{3/4}=\frac{3}{4}$	0.25
		$y^{1/2}=y^{1/4}=\frac{3}{4}$	0.25
(1,1)	1	$y^1=1.5$	0.5
		$y^{3/4}=1.3$	0.3
		$y^{1/2}=1.1$	0.1
		$y^{1/4}=0.99$	0.01
(1/2,1/2)	1	$y^1=1.66$	0.66
		$y^{3/4}=1.6$	0.6
		$y^{1/2}=1.57$	0.57
		$y^{1/4}=1.53$	0.53
		$y^{0.1}=1.4$	0.4

The 3-variable FD-PNN (Table 2) for linear true function approximation (e.g., $f\left(x_1,x_2,x_3\right)=x_1+x_2+x_3$ ) may involve one block of six neurons, FDE terms of all 1 and 2-combination derivative variables of the complete FDE, e.g.:

(13)

and:

(14)

Table 2. Approximation values of $f\left(x_1,x_2,x_3\right)=x_1+x_2+x_3$.

Data	Actual Value	Approximate Value	Absolute Error
(1,0,0)	1	$y_4^1=y_4^{3/4}=\frac{1}{2}$	0.5
		$y_4^{1/2}=y_4^{1/4}=\frac{1}{2}$	0.5
(0,1,0)	1	$y_4^1=y_4^{3/4}=\frac{1}{2}$	0.5
		$y_4^{1/2}=y_4^{1/4}=\frac{1}{2}$	0.5
(0,0,1)	1	$y_4^1=y_4^{3/4}=1$	0
		$y_4^{1/2}=y_4^{1/4}=1$
(1,1,0)	1	$y_4^1=1.125$	0.125
		$y_4^{3/4}=1.025$	0.025
		$y_4^{1/2}=1.873$	0.12
		$y_4^{1/4}=0.936$	0.063
(1,0,1)	1	$y_4^1=1.5$	0.5
		$y_4^{3/4}=1.368$	0.368
		$y_4^{1/2}=1.249$	0.249
		$y_4^{1/4}=1.1$	0.1
(1,1,1)	1	$y_4^1=1.6$	0.6
		$y_4^{3/4}=1.488$	0.488
		$y_4^{1/2}=1.26$	0.26
		$y_4^{1/4}=1.0755$	0.0755

Table 2. Approximation values of $f\left(x_1,x_2,x_3\right)=x_1+x_2+x_3$.

Data	Actual Value	Approximate Value	Absolute Error
(1,0,0)	1	$y_4^1=y_4^{3/4}=\frac{1}{2}$	0.5
		$y_4^{1/2}=y_4^{1/4}=\frac{1}{2}$	0.5
(0,1,0)	1	$y_4^1=y_4^{3/4}=\frac{1}{2}$	0.5
		$y_4^{1/2}=y_4^{1/4}=\frac{1}{2}$	0.5
(0,0,1)	1	$y_4^1=y_4^{3/4}=1$	0
		$y_4^{1/2}=y_4^{1/4}=1$
(1,1,0)	1	$y_4^1=1.125$	0.125
		$y_4^{3/4}=1.025$	0.025
		$y_4^{1/2}=1.873$	0.12
		$y_4^{1/4}=0.936$	0.063
(1,0,1)	1	$y_4^1=1.5$	0.5
		$y_4^{3/4}=1.368$	0.368
		$y_4^{1/2}=1.249$	0.249
		$y_4^{1/4}=1.1$	0.1
(1,1,1)	1	$y_4^1=1.6$	0.6
		$y_4^{3/4}=1.488$	0.488
		$y_4^{1/2}=1.26$	0.26
		$y_4^{1/4}=1.0755$	0.0755

We proceed to compute approximations for non-linear functions. We let $u: =f\left(x_1,x_2\right)$ be a function with square power variables, then we have:

$$F\left(x_1,x_2,u,\frac{{\partial }^{\beta }u}{\partial x_1^{\beta }},\frac{{\partial }^{\beta }u}{\partial x_2^{\beta }},\frac{{\partial }^{2\beta }u}{\partial x_1^{2\beta }},\frac{{\partial }^{2\beta }u}{\partial x_2^{2\beta }},\frac{{\partial }^{2\beta }u}{\partial x_1^{\beta }\partial x_2^{\beta }}\right)=0.$$

(15)

For example, for $\beta =1,$ we get [4]:

(16)

In general, for fractional power $\beta ,$ we have:

(17)

For example:

(18)

3.2. Modified Information Theory

In this section, we try to measure the learning of the neuron of the system in Figure 1. We wish to improve a applicable measure of the information we get from observing the appearance of an event having probability p. The approach depends on the probability of extinction, which describes by the fractional Poisson process as follows [28]:

$$P_{\beta }\left(N,y\right)= \frac{{\left(\sigma y\right)}^N }{N!} {\displaystyle \sum }_{n=0}^{\infty }\frac{\left(n+N\right)!}{n!} \frac{{\left(-\sigma y^{\beta }\right)}^n }{\text{Γ}\left(\beta \left(n+N\right)+1\right)},$$

(19)

where σ ϵ R is a physical coefficient, β ϵ (0,1]. Let N be the number of neurons, I be the average information and further that the source emits the symbols with probabilities P₁, P₂, ... , P_N, respectively such that P_i=P_β (i,y). Thus we may compute the total information as follows:

$$I={\displaystyle \sum }_{i=1}^N\left(N{P}_{i } \right)\log \left(\frac{1}{P_i}\right).$$

(20)

The last assertion is modified work due to Shannon [29]. For example, to compute the average information of the system with N=3, for the last fractional derivative in Table 3, we have:

$$\begin{array}{c}I={\displaystyle \sum _{i=1}^3}\left(N{P}_{i } \right)\log \left(\frac{1}{P_i}\right)=3P_1\log \left(\frac{1}{P_1}\right)+3P_{2 }\log \left(\frac{1}{P_2}\right)+3P_3\log \left(\frac{1}{P_3}\right) \\ \simeq 0.2408-0.09-0.051=0.051,\end{array}$$

(21)

where P_i converged to a hypergeometric function, which computed with the help of Maple.

Table 3. The approximation errors for $f\left(x_1,x_2\right)={(x_1+x_2)}^2$.

Data	Actual Value	Approximate Value	Absolute Error
(1,0)	2	$y_{10}^1=2$	0
		$y_{10}^{3/4}=1.834$	0.166
		$y_{10}^{1/2}=1.681$	0.319
		$y_{10}^{1/4}=1.5422$	0.4577
(0,1)	2	$y_{10}^1=2$	0
		$y_{10}^{3/4}=1.834$	0.166
		$y_{10}^{1/2}=1.681$	0.319
		$y_{10}^{1/4}=1.5422$	0.4577
(1,1)	2	$y_{10}^1=2.49$	0.49
		$y_{10}^{3/4}=2.04$	0.04
		$y_{10}^{1/2}=1.665$	0.335
		$y_{10}^{1/4}=1.336$	0.633

Table 3. The approximation errors for $f\left(x_1,x_2\right)={(x_1+x_2)}^2$.

Data	Actual Value	Approximate Value	Absolute Error
(1,0)	2	$y_{10}^1=2$	0
		$y_{10}^{3/4}=1.834$	0.166
		$y_{10}^{1/2}=1.681$	0.319
		$y_{10}^{1/4}=1.5422$	0.4577
(0,1)	2	$y_{10}^1=2$	0
		$y_{10}^{3/4}=1.834$	0.166
		$y_{10}^{1/2}=1.681$	0.319
		$y_{10}^{1/4}=1.5422$	0.4577
(1,1)	2	$y_{10}^1=2.49$	0.49
		$y_{10}^{3/4}=2.04$	0.04
		$y_{10}^{1/2}=1.665$	0.335
		$y_{10}^{1/4}=1.336$	0.633

4. Discussion

The presented 2-variable FD-PNN (Table 1) is able to approximate any linear function, e.g., the simple sum $f\left(x_1,x_2\right)=x_1+x_2.$ The comparison processes with respect to D-PNN (normal case) showed that the proposed method converged to the exact values rapidly. For example, the case (1,1) implied ABE=0.01 at $y^{1/4}.$ In this experiment, we let $b_0=1,w_1=w_2=1.$Figure 2 shows the approximation of the fractional derivative for the function $f\left(x_1,x_2\right)=x_1+x_2.$ The x-axis represents to the values when $x_1=x_2.$ It is clear that the interval of convergence is $\left[0.2,1\right].$ The endowed 3-variable FD-PNN (Table 2) is qualified to approximate any linear function e.g. simple sum $f\left(x_1,x_2,x_3\right)=x_1+x_2+x_3.$ The comparison procedure with respect to D-PNN displayed that the proposed method, of 3-variables, is converged swiftly to the exact values. For example, the case (1,1,0), with $w_2=3/2$ and (1,1,1), with $w_2=1$ yield ABE=0.063 and 0.0755 respectively at $y^{1/4}.$ Furthermore, Figure 3 shows the interval of convergence at $\left[0.3,1\right].$ Here, we let x₁ = x₂ = x₃. Comparable argument can be concluded from the non-linear case, where Table 3 computes approximation values, by utilizing FD-PNN. For example, the data (1,1) give the best approximation at $y^{3/4}$ when $w_{10}=\mathrm{1.5.}$ In Figure 4, the x-axis performs to the value when $x_1=x_2.$ Obviously, the interval of convergence is [0.4,2].

Figure 2. Selected fractional approximation derivative of f(x₁,x₂) = x_{1 +} x₂.

Figure 2. Selected fractional approximation derivative of f(x₁,x₂) = x_{1 +} x₂.

Figure 3. The fractional approximation y₄ of the function f(x₁,x₂,x₃) = x_{1 +} x_{2 +} x₃.

Figure 3. The fractional approximation y₄ of the function f(x₁,x₂,x₃) = x_{1 +} x_{2 +} x₃.

Figure 4. The fractional approximation y₁₀ of the function f(x_1,x₂) = (x₁ + x₂)².

Figure 4. The fractional approximation y₁₀ of the function f(x_1,x₂) = (x₁ + x₂)².

5. Conclusions

Based on GMDH-PNN (Figure 1) and modifying the work described in [4], we suggested a generalized D-PNN, called FD-PNN. The experimental results showed that the proposed method satisfies a quick approximation to the exact value comparison with the normal method. The generalization depended on the Riemann-Liouville differential operator. This method can be employed on data to describe modelling of complex systems. Next step, our aim is to modify this work by utilizing mixed D-PNN and FD-PNN, e.g. one can consider a function of the form:

$$F\left(x_1,x_2,u,\frac{\partial u}{\partial x_1},\frac{\partial u}{\partial x_2},\mathrm{...}\frac{{\partial }^{\beta }u}{\partial x_1^{\beta }},\frac{{\partial }^{\beta }u}{\partial x_2^{\beta }},\frac{{\partial }^{2\beta }u}{\partial x_1^{2\beta }},\frac{{\partial }^{2\beta }u}{\partial x_2^{2\beta }},\frac{{\partial }^{2\beta }u}{\partial x_1^{\beta }\partial x_2^{\beta }},\mathrm{...}\right)=0$$

(22)