Integral Neuron: A New Concept for Nonlinear Neuron Modeling Using Weight Functions. Creation of XOR Neurons

Abstract

In the present study, an extension of the idea of dynamic neurons is proposed by replacing the weights with a weight function that is applied simultaneously to all neuron inputs. A new type of artificial neuron called an integral neuron is modeled, in which the total signal is obtained as the integral of the weight function. The integral neuron enhances traditional neurons by allowing the signal shape to be linear and nonlinear. The training of the integral neuron involves finding the parameters of the weight function, where its functional values directly influence the total signal in the neuron’s body. This article presents theoretical and experimental evidence for the applicability and convergence of standard training methods such as gradient descent, Gauss–Newton, and Levenberg–Marquardt in searching for the optimal weight function of an integral neuron. The experimental part of the study demonstrates that a single integral neuron can be trained on the logical XOR function—something that is impossible for single classical neurons due to the linear nature of the summation in their bodies.

1. Introduction

The key characteristics of dynamic neural networks (DNNs) are their changing structure, temporal dependency, greater expressive power, and better adaptability than static artificial neural networks (ANNs). Regarding their changing structure, it should be noted that the number of neurons, the connections between them, and even the entire architecture of a DNN can change during training or operation [1,2,3]. Their temporal dependency is linked to the fact that these DNNs can account for previous states and predict future events, making them particularly useful for tasks in meteorology [4,5,6,7], financial market analysis [8,9,10,11], speech recognition [12,13,14,15], and even robot control [16,17,18,19]. The greater expressive power lies in the fact that DNNs can represent more complex functions and patterns than static ANNs, while their better adaptability allows them to adjust to changing conditions and learn from new data more effectively. However, the strengths of DNNs also contribute to some challenges in their use. For example, designing and training DNNs is more complex compared to static ANNs, and both the training and subsequent operation often require more computational resources.

This article presents various stages of modeling a new type of neuron called integral neurons, which use weight functions instead of fixed weights during the training process, as well as the implementation of an integral XOR neuron that can be implemented with different weight and transfer functions. The main sections of the article, following the introduction, are as follows:

• In the second part of the study, various DNNs are examined. The idea is proposed that a new type of dynamic behavior can be created by replacing the static weights determined during the training of ANNs with weight functions, allowing for the creation of a nonlinear signal within the neuron.
• In the third part, definitions are proposed for a weight function, which is used instead of static weights within the neuron, and an integral neuron, featuring an integral function in its body that uses the weight function. Evidence is provided for the applicability of training methods such as gradient descent, Gauss–Newton, and Levenberg–Marquardt in the search for the optimal weight function.
• The fourth part—Discussion—examines the similarities and differences between classical neurons and the newly proposed integral neurons and provides directions for potential developments in the topic of artificial integral structures. Definitions are provided for integral and integral–classical neural networks, among others.
• In the experimental fifth part, the possibility of creating a single integral neuron that solves the XOR logical function, which is unachievable by standard artificial neurons, is explored. Theoretical and experimental evidence for the creation of integral XOR neurons, trained with several different weight and transfer functions, is presented. The MATLAB v2018a scripts used for creating the integral XOR neurons are provided in Appendix A.

2. Dynamic Artificial Neural Networks

The main types of DNNs are Recurrent Neural Networks (RNNs), Long Short-Term Memory (LSTM) networks, Gated Recurrent Units (GRUs), Time-Delay Neural Networks (TDNNs), Echo State Networks (ESNs), and Temporal Convolutional Networks (TCNs) [20,21].

2.1. Recurrent Neural Networks

RNNs contain feedback loops in their architecture which allow information from previous time steps to be retained and used for current computations. These loops cyclically pass information through time, as RNNs maintain an internal state $h_t$, which is updated at each time step $t$. This state is used to store information from previous inputs.

In standard feedforward ANNs, neurons in one layer transmit signals only to the neurons in the next layer. The weight matrices $W$ determine the connections between the layers, and the states of the network (the outputs from the neurons) are

$$h=g\left(Wx+b\right),$$

(1)

where

• $x$ is the input vector;
• $b$ is the bias vector (thresholds);
• $g$ is the transfer function.

In contrast, RNNs have an additional aspect—the feedback over time. At each time step, RNNs not only take into account the current input $x_t$, but also the previous state $h_{t-1}$ [22,23]. This allows the network to have memory and retain information across time steps. At each time step $t$, the RNN performs the following computations:

$$h_t=g\left(W_h{h}_{t-1}+W_x{x}_t+b\right),$$

(2)

where

• $h_t$ is the hidden state of the network at time step $t$;
• $h_{t-1}$ is the hidden state of the network at the previous time step;
• $x_t$ is the current input;
• $W_h$ and $W_x$ are the weight matrices.

The weight matrix $W_h$ processes the previous state $h_{t-1}$ and provides the information accumulated from prior time steps. This is the key element that gives RNNs the ability to remember past information. The weight matrix $W_x$ processes the current input $x_t$, which is analogous to what happens in standard ANNs. It handles the processing of new information that enters at each time step. This structure allows RNNs to combine new information with the accumulated memory from previous time steps, making it possible to recognize temporal dependencies. It should be noted that deep RNNs include multiple recurrent layers, with each layer receiving input from the previous one. This enables the network to capture more complex dependencies in the data. If there are two recurrent layers, the computations will be performed with the following equations, characterizing the states (outputs) of the layers:

$$\left\{\begin{array}{l}{h}_t^{\left(1\right)}=g\left(W_h^{\left(1\right)}{h}_{t-1}^{\left(1\right)}+W_x^{\left(1\right)}{x}_t+b^{\left(1\right)}\right)\\ h_t^{\left(2\right)}=g\left(W_h^{\left(2\right)}{h}_{t-1}^{\left(2\right)}+W_x^{\left(2\right)}{h}_t^{\left(1\right)}+b^{\left(2\right)}\right)\end{array}\right.,$$

(3)

where

• $h_t^{\left(1\right)}$ and $h_t^{\left(2\right)}$ are the hidden states of the first and second layers, respectively;
• $W_h^{\left(1\right)}$, $W_x^{\left(1\right)}$, $W_h^{\left(2\right)}$, and $W_x^{\left(2\right)}$ are the weight matrices for the respective layers;
• $b^{\left(1\right)}$ and $b^{\left(2\right)}$ are the bias vectors (containing thresholds) for the respective layers;
• $g$ is the transfer function.

Examples of the dynamic capabilities of RNNs can be found in [24,25,26,27].

2.2. Long Short-Term Memory Networks

LSTM networks are a type of RNN specifically designed to address the vanishing gradient problem in RNNs by introducing memory cells and gates that control the flow of information. The main components of LSTM are the forget gate, input gate, memory cell, and output gate, with dynamic behavior achieved through more complex equations that involve memory cells and gates controlling the flow of information [28,29]:

$$\left\{\begin{array}{l}{f}_t=g\left(W_f{x}_t+U_f{h}_{t-1}+b_f\right)\\ i_t=g\left(W_i{x}_t+U_i{h}_{t-1}+b_i\right)\\ o_t=g\left(W_o{x}_t+U_o{h}_{t-1}+b_o\right)\\ \widetilde{c_t}=g\left(W_c{x}_t+U_c{h}_{t-1}+b_c\right)\\ c_t=f_t⨀c_{t-1}+i_t⨀\widetilde{c_t}\\ h_t=o_t⨀g\left(c_t\right)\end{array}\right..$$

(4)

Here,

• $f_t$ is the state of the forget gate;
• $i_t$ and $o_t$ are the input and output gates, respectively;
• $\widetilde{c_t}$ is the proposed new value for the cell, i.e., the cell gate activation vector;
• $c_t$ is the current cell state;
• $h_t$ is the current hidden state.

In all the presented equations, the subscript $t$ indicates the time step, $g$ is a sigmoid-like transfer function (most commonly the hyperbolic tangent), and $\odot$ represents the Hadamard product—a binary operation that takes two matrices of the same dimensions and returns a matrix of element-wise multiplications. These equations demonstrate how LSTM networks can retain and manipulate information over long time periods by using complex memory management mechanisms. In their work, Zhumei Wang et al. use the dynamic characteristics of LSTM networks for accurate traffic forecasting [30], while Ruijie Huang et al. base their model on LSTM dynamics to create a system fed with historical data from 4000 previous days, predicting the production efficiency of a carbonate reservoir well for the next 500 days [31]. The use of dynamics in LSTM models can be explored in many other studies [32,33,34,35].

2.3. Gated Recurrent Units

GRUs are improved versions of earlier LSTM models. They have a simplified structure compared to LSTM, as they use fewer gates and parameters. This makes them easier to train and often faster than LSTM, especially when working with smaller datasets. The GRU cell includes two main gates: the update gate and the reset gate. The update gate helps the cell decide how much of the previous information will be retained in the new state. It combines the functions of the input gate and the forget gate in LSTM, thus controlling the extent to which information will be carried over to future time steps. The reset gate determines how much of the previous information should be forgotten. This gate allows the model to discard irrelevant information, which can be especially useful when modeling time series, where earlier data points may not always be important for understanding later points.

There are different variations of GRUs. For instance, simplified versions of GRUs like the Minimal Gated Unit can be used, where the gating mechanisms are reduced to a single main mechanism for controlling input and forget information [36,37]. Another variation is the Coupled GRU, where there is a dependency between the reset and update gates, rather than using them as completely independent elements [38]. GRU-Ds are modifications with a bidirectional architecture designed to handle time series with missing data [39], among others. In the system of Equation (5), the operation and key components of the mathematical model of the fully gated version of GRUs are presented [40]:

$$\left\{\begin{array}{l}{z}_t=\sigma \left(W_z{x}_t+U_z{h}_{t-1}+b_z\right)\\ r_t=\sigma \left(W_r{x}_t+U_r{h}_{t-1}+b_r\right)\\ \widetilde{h_t}=\varphi \left(W_h{x}_t+U_h(r_t⨀h_{t-1})+b_h\right)\\ h_t=\left(1-z_t\right)⨀h_{t-1}+z_t⨀\widetilde{h_t}\end{array}\right..$$

(5)

Here,

• $x_t$ and $h_t$ are the input and output vectors, respectively;
• $\widetilde{h_t}$ is the candidate activation vector;
• $z_t$ and $r_t$ are the states of the update and reset gates, respectively.

The transfer functions in this case, $\sigma$ and $\varphi$, are the logistic function and hyperbolic tangent, respectively. The dynamic nature of the architecture and functionality of GRUs makes them attractive tools for various tasks. Examples of their use can be found in numerous studies [41,42,43,44].

2.4. Time-Delay Neural Networks

The main idea behind TDNNs is the introduction of a time delay for the input data, allowing the network to access historical (previous) values for a certain number of time steps. This specific feature makes them particularly suitable for tasks involving time sequences or time-dependent data, such as speech recognition, audio signal processing, and time series analysis. The input data to a TDNN are fed in as consecutive time slices, so at a given time $t$, the neuron will receive inputs not only for the current moment but also for previous moments $(t-1), (t-2)$, etc. The output $y\left(t\right)$ of a neuron in a TDNN at time $t$ can be expressed as a sum of the input data $x\left(t\right)$ at the current and previous time steps, multiplied by the corresponding weights $w_i$ and thresholds $b$:

$$y\left(t\right)=g\left(\sum _{i=0}^d{w}_{i}x\left(t-i\right)+b\right).$$

(6)

If multiple layers are considered, the output of a neuron from the $l$-th layer is given by

$$y_j^{\left(l\right)}\left(t\right)=g\left(\sum _{i=0}^d{w_{ij}}^{\left(l\right)}{y}_j^{(l-1)}\left(t-i\right)+b_j^{\left(l\right)}\right),$$

(7)

where $y_j^{(l-1)}\left(t-i\right)$ is the outputs from the neurons of the previous layers, $b_j^{\left(l\right)}$ is the corresponding biases, and $g$ is the transfer function. Examples of the use of TDNNs in solving problems related to speech recognition, capturing dependencies between sounds and their temporal structures, as well as audio signal processing, can be found in [45,46,47,48,49,50].

2.5. Echo State Networks

ESNs are a specific type of DNN primarily used for tasks involving time sequences. They belong to the broader class of reservoir computing and represent an alternative approach to modeling temporal dependencies in data without the heavy optimization of internal weights, as observed in classical RNN models. In practice, ESNs rely on a reservoir of neurons, which is a large, randomly connected recurrent network, where input data are transformed into a more complex, high-dimensional space using nonlinear connections. The key aspect of this type of DNN is that the internal weights in the reservoir are not trained through standard backpropagation algorithms but are randomly initialized and remain fixed during training. Unlike standard RNNs, in ESNs, only the weights of the output layer are trained. These weights connect the reservoir to the output neurons, a feature that greatly simplifies the training process, as only the optimization of linear connections is required, which can be achieved with simple linear regression or other straightforward methods.

The mathematical model of ESNs defines the state of the reservoir $x\left(t\right)$ as being updated at each time step $t$ by the following equation:

$$x\left(t\right)=g\left(W_{res}x\left(t-1\right)+W_{in}u\left(t\right)+b\right),$$

(8)

where $W_{res}$ is the weight matrix of the reservoir, $W_{in}$ is the weight matrix connecting the inputs to the reservoir, and $u\left(t\right)$ and $b$ are the matrices of input data and biases, respectively. The outputs of the ESNs are expressed as a linear combination of the current state of the reservoir:

$$y\left(t\right)=W_{out}x\left(t\right),$$

(9)

where $W_{out}$ represents the output weights, which are optimized during training.

In [51], the authors use ESNs for a new approach to time series data prediction, and Huanling Hu et al. propose a hybrid model incorporating this type of network for forecasting wind speed at the Sotavento Wind Farm in Galicia, northwest Spain [52]. In this model, ESNs are used for prediction in each individual sub-series. Other interesting applications of ESNs can be explored in [53,54,55].

2.6. Temporal Convolutional Networks

TCNs are designed for tasks related to processing temporal sequences, based on the principles of convolutional neural networks (CNNs). Their convolution ensures that the output at time $t$ depends only on current and previous values over time, but not on future values. The use of convolutional layers allows for greater parallelization and faster training, as at any given moment, a TCN examines multiple previous time steps simultaneously. This quality makes them more efficient than traditional recurrent networks, where processing each step depends on the previous one. Additionally, TCNs use dilated convolution, which helps extend the receptive field of the convolutional filters without necessarily increasing the number of parameters. This allows the TCN to “see” longer time dependencies while remaining efficient. To ensure that the output at time $t$ depends only on current and previous values over time and not on future values, the convolution is structured so that the output $y\left(t\right)$ depends only on the current and previous inputs $x\left(t\right),x(t-1),\dots ,x(t-k+1)$:

$$y\left(t\right)=g\left(\sum _{i=0}^{k-1}{w}_{i}x\left(t-i\right)+b\right),$$

(10)

where $w_i$ and $b$ are the filter weights and biases, respectively, and $g$ is the transfer function. TCNs are highly effective for forecasting values from time series, such as financial data predictions [56,57] or climate conditions [58,59]. Additionally, they can be used for tasks related to text processing [60,61] and natural language processing [62,63], as well as for problems involving speech, audio, and video processing [64,65].

2.7. Conclusions Regarding Dynamic Artificial Neural Networks

The dynamic nature of the DNNs discussed is expressed in their ability to process and model temporal dependencies and sequences in the data. Unlike static ANNs, which accept a fixed set of input data and produce outputs without considering the order or temporal dependencies between input values, DNNs have internal mechanisms that allow for the processing of data that changes over time. Thus, during the training process, DNNs update their parameters (weights and biases) based not only on the current input data but also on the errors accumulated over time. This means that each iteration of training takes into account the temporal sequence of the data.

However, is it possible to add even more to the dynamic aspect? Is it possible, during training, for networks to be equipped with linear or nonlinear weight functions instead of a weight vector, thereby allowing the signal within the neuron’s body and the neuron’s output to change the way they are formed? We provide a positive answer to these questions in the following sections of this article.

3. Integral Neurons with Weight Functions: Training Algorithms for Integral Neurons

In this section, we explore the possibility of creating a new type of dynamic neuron with dynamically changing weights through the definition of weight functions. We propose definitions for the concepts of weight function and integral neuron. The training methods of gradient descent, Gauss–Newton, and Levenberg–Marquardt are examined, with algorithms and theoretical evidence presented for their applicability in finding the optimal weight function for an integral neuron.

3.1. Problem Definition

Let us consider a single artificial neuron influenced by $n$ stimuli $x_1, x_2, \dots ,x_n$, where $x_i\in R$ (Figure 1).

In its body, a total cumulative signal is formed from all inputs:

$$S=\sum _{i=0}^n{w}_i{x}_i, w_i\in R \forall i=1,2,\dots ,n$$

(11)

where the standard and common procedure of including the neuron’s threshold $b$ in the total sum is implemented:

$$x_0=1, w_0=b.$$

(12)

Let a task be set in which the neuron must be trained to approximate the target function $H$ using a set of $m$ training samples:

$${D=\left\{[x_{j0},x_{j1},\dots ,x_{jn}; h_j]\right\}}_{j=1}^m,\hspace{1em}{x}_{j0}=1, \forall j=1, 2,\dots , m.$$

(13)

In dataset (13), $\left(x_{j0},x_{j1},\dots ,x_{jn}\right)$ is the input set, and $h_j$ is its corresponding functional value of $H$, i.e.,

$$h_j=H\left(x_{j0},x_{j1},\dots x_{jn}\right), j=1, 2,\dots , m.$$

(14)

Classical procedures related to training the neuron are aimed at finding the weight vector

$$\overrightarrow{W}=\left(w_0, w_1,\dots , w_n\right), {\forall w}_i=const\in R, i=1, 2,\dots ,n,$$

(15)

for which the error function

$$E(w_0,w_1,\dots , w_n)={\displaystyle \frac{1}{m}}\sum _{j=1}^m{\left(h_j-g\left(x_{j0}, x_{j1}, \dots , x_{jn}, w_1, w_2,\dots , w_n\right)\right)}^2$$

(16)

is at its minimum.

Once the desired performance of the neuron is achieved, the specific weight set (15) that achieves this remains unchanged (${\forall w}_i\in R$), unless the neuron is retrained. Thus, the artificial neuron will respond to any subsequent stimuli on its inputs with the constant weight vector (15) obtained after training.

The question we pose is the following: Is it possible to conduct training that seeks not a weight vector of constants but a function,

$$w=w\left(x_i\right), i=1, 2,\dots , n,$$

(17)

where the weight for each input will vary each time, depending on the value of the corresponding stimulus (Figure 2)?

Replacing static weights with a weight function would create a new type of artificial neuron and neural structure, where the signal shape within the neuron’s body could now be nonlinear. This enables the development of new, more effective solutions for various tasks. For instance, the fundamental unsolved problem of creating a single XOR neuron with traditional techniques is resolved with these new methods and is presented later in this article.

If we reconsider the standard neuron shown in Figure 2, we will notice that all terms in the sum

$$S=\sum _{i=0}^n{w}_i{x}_i=x_0{w}_0+x_1{w}_1+\cdots +x_n{w}_n$$

(18)

can be viewed as the areas of rectangles (Figure 3).

Let the variational series obtained from the input values of the neuron be

$$x_0^{\prime }=x_{min}, x_1^{\prime }, x_2^{\prime }, \dots , x_n^{\prime }=x_{max}$$

(19)

Then, the total signal (18) within the neuron’s body can be represented as the sum of the areas of adjacent rectangles (Figure 4).

Given that

$$\sum _{i=0}^n{x}_i^{\prime }=\sum _{i=0}^n{x}_{i },$$

(20)

and if $∆x_i=x_i-x_{i-1},i=1,2,\dots ,n$ and $\lambda =\max \left\{∆x_1,∆x_2,\dots ,∆x_n\right\}\to 0$, then

$$S={\int }_0^{{\displaystyle {\sum }_{i=0}^n}{x}_i}w\left(t\right)dt.$$

(21)

As a result, along the axon of a standard neuron with transfer function $g$, the signal has a value infinitely close to

$$Out=g\left({\int }_0^{{\displaystyle {\sum }_{i=0}^n}{x}_i}w\left(t\right)dt\right).$$

(22)

Given the training set (13) ${D=\left\{[x_{j0}=1,x_{j1},\dots ,x_{jn}; h_j]\right\}}_{j=1}^m$, the system of errors for each of the samples is

$$\left\{\begin{array}{l}{e}_1=h_1-g\left({\int }_0^{{\displaystyle {\sum }_{i=0}^n}{x}_{1i}}w\left(t\right)dt\right)\\ e_2=h_2-g\left({\int }_0^{{\displaystyle {\sum }_{i=0}^n}{x}_{2i}}w\left(t\right)dt\right)\\ \dots \\ e_m=h_m-g\left({\int }_0^{{\displaystyle {\sum }_{i=0}^n}{x}_{mi}}w\left(t\right)dt\right)\end{array}\right..$$

(23)

The mean squared error (MSE) of the network is given by

$$MSE={\displaystyle \frac{1}{m}}\sum _{j=1}^m{e}_j^2,$$

(24)

i.e.,

$$MSE={\displaystyle \frac{1}{m}}\sum _{j=1}^m{\left(h_j-g\left({\int }_0^{{\displaystyle {\sum }_{i=0}^n}{x}_{ji}}w\left(t\right)dt \right)\right)}^2.$$

(25)

The last expression shows that the task of training algorithms for the ANN in this case should not be focused on finding a specific set of weights, as is typical in standard approaches. Instead, the focus should shift towards finding the function $w\left(t\right)$ that minimizes the error function (25). Given the representation of the MSE, it becomes clear that we are, in fact, searching for the minimum of the function

$$E=\sum _{j=1}^m{\left(h_j-g\left({\int }_0^{{\displaystyle {\sum }_{i=0}^n}{x}_{ji}}w\left(t\right)dt \right)\right)}^2.$$

(26)

Many training algorithms utilize derivatives of the error function, which in turn imposes the condition of differentiability on the integral function involved in (26). Thus, to ensure the correct calculation of derivatives in training algorithms, it is necessary for the integrand function $w\left(t\right)$ to possess a certain degree of smoothness. Specifically, the differentiability of $w\left(t\right)$ enables the application of methods such as gradient descent, Gauss–Newton, and Levenberg–Marquardt, which we will demonstrate in this study. For this purpose, we consider it necessary to introduce concepts for a continuously differentiable weight function and an integral neuron.

Let $C^k$ be the class of $k$-times differentiable functions, and let $N_0$ be the set of natural numbers with zero included, i.e., $N_0=N\cup \left\{0\right\}$.

Definition 1

(Weight function of a neuron). A continuously differentiable real function $w\left(t\right)\in C^k, k\in N_0$, defining the cumulative signal in the body of an artificial neuron by the equation

$$S={\int }_0^{{\displaystyle {\sum }_{i=0}^n}{x}_i}w\left(t\right)dt,$$

where $x_0,x_1,x_2,\dots ,x_n$ are the input stimuli, will be called the weight function for the respective neuron.

Definition 2

(Integral neuron). An artificial neuron that uses a weight function $w\left(t\right)\in C^k, k\in N_0$ to form the signal within its body will be called an integral neuron of class $(k+1)$.

Let us note that, due to the nature of integration, the class of the integral function $S$, which describes the signal within the neuron’s body, is one degree higher than that of the weight function. This increase draws our attention to the fact that the integration process provides an additional level of smoothness. For example, if a twice-differentiable weight function is chosen, the signal within the neuron’s body will be three times differentiable.

3.2. Algorithm for Finding the Optimal Weight Function Using Gradient Descent of the Error

The unusual aspect of implementing our proposed idea is that the training algorithm seeks not a vector of numerical weight values that minimizes the error but rather the type and parameters of a weight function. In the context of modeling the weight function, the coefficients of the weight function are also referred to as parameters.

When using the error minimization algorithm (Figure 5) with gradient descent, the following stages are followed:

• Initialization: At this stage, an initial function $w^{\left(0\right)}\left(t\right)$ is selected: this can be any continuous and parametric linear or nonlinear function. An initial value is set for the parameters of $w\left(t\right)$ (for example, for a polynomial or exponential function).
• Error calculation: For each $j$-th sample in the training set (13) ${D=\left\{[x_{j0}=1,x_{j1},\dots ,x_{jn}; h_j]\right\}}_{j=1}^m$, the neuron’s output is calculated as

  $$y_j=g\left({\int }_0^{{\displaystyle {\sum }_{i=0}^n}{x}_{ji}}w\left(t\right)dt\right),$$

  (27)

  the errors are computed as

  $$e_j=h_j-y_j, \forall j=1, 2,\dots , m,$$

  (28)

  and the MSE is formed as

  $$E={\displaystyle \frac{1}{m}}\sum _{j=1}^m{e}_j^2.$$

  (29)
• Gradient calculation and parameter update of the weight function: Unlike the traditional approach, at this stage, the derivative of the error is calculated not with respect to weights and biases, but with respect to the parameters of the weight function $w\left(t\right)$. For example, in the case of a polynomial form for $w\left(t\right)$, derivatives are calculated with respect to the polynomial’s parameters. This means we are actually searching for the optimal parameters of the weight function that minimize the error (29). At each step $k$, the parameter update of $w\left(t\right)$ is performed using the calculated gradient:

  $$w^{(k+1)}\left(t\right)=w^{\left(k\right)}\left(t\right)-\eta {\displaystyle \frac{\partial E}{\partial w\left(t\right)}},$$

  (30)

  where $\eta >0\in R$ is the descent step—a training hyperparameter, also known as the learning rate. The descent step determines how quickly the algorithm will move in the direction of the steepest slope of the error function. The larger its value, the faster the parameters will be updated, but the higher the risk of missing the minimum. On the other hand, a very small step can make the training process very slow, as a small $\eta$ value will require more iterations. The value of $\eta$ can be set by the user implementing the algorithm or can be built into software solutions, with an option for adaptive adjustment during training.
• Iterations: Steps 2 and 3 are repeated until one of the following conditions is met:
  • A local or global minimum of the error function is found;
  • One of the termination conditions set in the software implementation for optimization is met (e.g., maximum number of epochs reached, specified number of objective function calls exceeded, error plateau identified for a specified number of iterations, etc.).

Upon completion of the algorithm, the desired weight function is ${w\left(t\right)=w}^{\left(l\right)}\left(t\right)$, where $l$ is the number of the last iteration performed.

3.2.1. Example Application of the Algorithm

To clarify the proposed algorithm, let us consider a polynomial weight function $w\left(t\right)$ for a neuron whose transfer is achieved through a hyperbolic tangent. The choice of a polynomial weight function in this case is justified, as it is differentiable and easy to integrate. Additionally, according to the Weierstrass Approximation Theorem, any continuous function on a closed interval can be approximated by a polynomial [66], such as a Taylor series.

Given a polynomial weight function of degree $q$,

$$w\left(t\right)=a_0+a_{1}t+a_2{t}^2+\cdots +a_q{t}^q, \forall a_p\in R, p=0, 1, 2,\dots , q.$$

(31)

Following the described algorithm, the following stages should be completed:

• Initialization:

Initial values for the parameters $a_0,a_1,a_2,\dots ,a_q$ are selected.

2.

Error calculation:

For each sample from the training set, the output of the neuron is

$$y_j=\mathrm{t}\mathrm{a}\mathrm{n}\mathrm{h}\left({\int }_0^{{\displaystyle {\sum }_{i=0}^n}{x}_{ji}}w\left(t\right)dt\right).$$

(32)

After substituting the integrand with the polynomial (31) and solving the integral, we can easily reach the representation

$$y_j=\mathrm{t}\mathrm{a}\mathrm{n}\mathrm{h}\left(\sum _{p=0}^q{a}_p{\displaystyle \frac{{\left({\displaystyle {\sum }_{i=0}^n}{x}_{ji}\right)}^{p+1}}{p+1}}\right),$$

(33)

which we use to calculate the errors (28) and (29), which become

$$e_j=h_j-\mathrm{t}\mathrm{a}\mathrm{n}\mathrm{h}\left(\sum _{p=0}^q{a}_p{\displaystyle \frac{{\left({\displaystyle {\sum }_{i=0}^n}{x}_{ji}\right)}^{p+1}}{p+1}}\right), \forall j=1,2,\dots ,m$$

(34)

and

$$E={\displaystyle \frac{1}{m}}\sum _{j=1}^m{\left(h_j-\mathrm{t}\mathrm{a}\mathrm{n}\mathrm{h}\left(\sum _{p=0}^q{a}_p{\displaystyle \frac{{\left({\displaystyle {\sum }_{i=0}^n}{x}_{ji}\right)}^{p+1}}{p+1}}\right)\right)}^2.$$

(35)

3.

Gradient calculation and parameter update of the weight function:

At this stage, the derivatives of the error (35) are calculated with respect to each of the parameters $a_p$ of the function $w\left(t\right)$:

$${\displaystyle \frac{\partial E}{\partial a_p}}=-{\displaystyle \frac{2}{m}}\sum _{j=1}^m{e}_j\left(1-y_j^2\right){\displaystyle \frac{x_j^{p+1}}{p+1}}.$$

(36)

The coefficients of the weight function are updated as follows:

$$a_p\leftarrow a_p-\eta {\displaystyle \frac{\partial E}{\partial a_p}}, \forall p=1, 2, \dots , q,$$

(37)

where $\eta >0\in R$ is the descent step.

4.

Iterations:

Steps 2 and 3 are repeated until a set of parameters $a_0, a_1,a_2,\dots ,a_q$ of the weight function is reached, for which the error meets the acceptable requirements.

3.2.2. Convergence of the Algorithm

Let us denote the parameters of the weight function and the gradient of the error function as column vectors:

$$a={\left[\begin{array}{c}{a}_0\\ a_1\\ a_2\\ ⋮\\ a_q\end{array}\right]}_{\left(q+1\right)\times 1},$$

(38)

and

$${\displaystyle \frac{\partial E}{\partial w\left(t\right)}}={\left[\begin{array}{c}{\displaystyle \frac{\partial E}{\partial a_0}}\\ {\displaystyle \frac{\partial E}{\partial a_1}}\\ {\displaystyle \frac{\partial E}{\partial a_2}}\\ ⋮\\ {\displaystyle \frac{\partial E}{\partial a_q}}\end{array}\right]}_{\left(q+1\right)\times 1}.$$

(39)

The possibility of finding the optimal function $w\left(t\right)$, for which the error function $E\left(w\left(t\right)\right)$ is minimal, is demonstrated by the following convergence theorem for the proposed algorithm.

Theorem 1

(Convergence of the Gradient Descent Method for Optimizing the Weight Function). Let an integral neuron of at least first class with a smooth transfer function $g$ be given. Let $a$ be the vector containing the parameters of the weight function $w\left(t\right)$, and ${\displaystyle \frac{\partial E}{\partial w\left(t\right)}}$ be the gradient of the error function.

Then, if the parameter update for $w\left(t\right)$ is performed through an optimization algorithm based on gradient descent of the mean squared error function $E\left(w\left(t\right)\right)$ with step $\eta >0$,

$$a^{\left(k+1\right)}\left(t\right)=a^k\left(t\right)-\eta {\displaystyle \frac{\partial E}{\partial w\left(t\right)}},$$

then the algorithm is convergent and brings the parameters of the function $w\left(t\right)$ closer to values that minimize $E\left(w\left(t\right)\right)$.

Proof of Theorem 1.

In the first stage, we will demonstrate the differentiability of the error function and the applicability of the gradient descent method.

Let an integral neuron (Figure 6) of at least first class with a smooth transfer function $g$ be given, which is trained with examples from the set ${D=\left\{[x_{j0}=1,x_{j1},\dots ,x_{jn}; h_j]\right\}}_{j=1}^m$.

Since the mean squared error function is represented as

$$E\left(w\left(t\right)\right)=\sum _{j=1}^m{\left(h_j-g\left({\int }_0^{{\displaystyle {\sum }_{i=0}^n}{x}_{ji}}w\left(t\right)dt\right)\right)}^2$$

(40)

it is also differentiable, as it is a composition of differentiable functions that comprise it. Therefore, the derivative ${\displaystyle \frac{\partial E}{\partial w\left(t\right)}}$ exists and is continuous for every value of $t$. Then, given the bounded nature of $E\left(w\left(t\right)\right)$ (the function is bounded below: $E\left(w\left(t\right)\right)\ge 0$), it follows that an optimization algorithm using gradient descent on this function can be applied.

In the second stage, we will demonstrate the reduction in the error.

$$a^{\left(k+1\right)}\left(t\right)=a^k\left(t\right)-\eta {\displaystyle \frac{\partial E}{\partial w\left(t\right)}},$$

(41)

At each iteration, the algorithm updates the weight function $w\left(t\right)$.

Here, $\eta >0$ is the descent step. Let $w^{\left(k\right)}\left(t\right)$ be the value of the weight function involved in the error (40) at the end of the $k$-th iteration. Then, its next value will be

$$w^{\left(k+1\right)}\left(t\right)=w^k\left(t\right)-\eta {\displaystyle \frac{\partial E}{\partial w\left(t\right)}} {|}_{w=w^k\left(t\right)}.$$

(42)

We will show that

$$E\left(w^{k+1}\left(t\right)\right)<E\left(w^k\left(t\right)\right).$$

(43)

Let us consider the first-order Taylor series expansion of the error function.

$$E\left(w^{k+1}\left(t\right)\right)\approx E\left(w^k\left(t\right)\right)+\left(w^{k+1}\left(t\right)-w^k\left(t\right)\right){\displaystyle \frac{\partial E}{\partial w\left(t\right)}} {|}_{w=w^k\left(t\right)}.$$

(44)

If we substitute $w^{k+1}\left(t\right)$ from the parameter update in Equation (42) into (44), we obtain

$$E\left(w^{k+1}\left(t\right)\right)\approx E\left(w^k\left(t\right)\right)-\eta {\left({\displaystyle \frac{\partial E}{\partial w\left(t\right)}} {|}_{w=w^k\left(t\right)}\right)}^2.$$

(45)

Since $\eta >0$ and ${\left({\displaystyle \frac{\partial E}{\partial w\left(t\right)}} {|}_{w=w^k\left(t\right)}\right)}^2>0$, it follows exactly what we intended to prove:

$$E\left(w^{k+1}\left(t\right)\right)<E\left(w^k\left(t\right)\right).$$

(46)

It is important to note the following: If the function $E\left(w\left(t\right)\right)$ is convex, the algorithm will approach the global minimum. Otherwise, the convergence may be towards a local minimum; however, the algorithm will still be convergent and will find the necessary parameters of the weight function $w\left(t\right)$ at which this minimum is achieved. □

3.3. Algorithm for Finding the Optimal Weight Function Using the Gauss–Newton Method

Applying the Gauss–Newton method within our framework of using a weight function instead of static weights still requires going through the previously described steps 1–4. The first three steps remain the same, and Equations (13) and (27)–(29) are once again satisfied (Figure 7). The new and key aspect is the update of the weight function parameters

$$a={\left[\begin{array}{c}{a}_0\\ a_1\\ a_2\\ ⋮\\ a_q\end{array}\right]}_{\left(q+1\right)\times 1},$$

(47)

which is now expressed through the transition

$$a \leftarrow a-{\left(J^{T}J\right)}^{-1}{J}^{T}e.$$

(48)

In the transition in (48), $J$ is the Jacobian matrix, containing the partial derivatives of the error $E\left(w\left(t\right)\right)$ with respect to each parameter of $w\left(t\right)$:

$$J={\left[\begin{array}{cc}\begin{array}{c}\begin{array}{ccc}{\displaystyle \frac{\partial e_1}{\partial a_0}}& {\displaystyle \frac{\partial e_1}{\partial a_1}}& \dots \\ {\displaystyle \frac{\partial e_2}{\partial a_0}}& {\displaystyle \frac{\partial e_2}{\partial a_1}}& \dots \\ ⋮& ⋮& \ddots \end{array}\\ \begin{array}{ccc}{\displaystyle \frac{\partial e_m}{\partial a_0}}& {\displaystyle \frac{\partial e_m}{\partial a_1}}& \dots \end{array}\end{array}& \begin{array}{c}\begin{array}{c}{\displaystyle \frac{\partial e_1}{\partial a_q}}\\ {\displaystyle \frac{\partial e_2}{\partial a_q}}\end{array}\\ ⋮\\ {\displaystyle \frac{\partial e_m}{\partial a_q}}\end{array}\end{array}\right]}_{m\times (q+1)},$$

(49)

and

$$e={\left[\begin{array}{c}{e}_1\\ e_2\\ e_3\\ ⋮\\ e_m\end{array}\right]}_{m\times 1}$$

(50)

is the vector containing all errors as each training sample passes through the neuron at each iteration. Here, $m$ is the number of training samples, and $q+1$ is the number of parameters of the weight function.

Theorem 2

(Convergence of the Gauss–Newton Method for Optimizing the Weight Function). Let an integral neuron of at least second class with a smooth transfer function $g$ be given. Let $a$ be the vector containing the parameters of the weight function, $J$ be the Jacobian matrix of the error function $E\left(w\left(t\right)\right)$, and $e$ be the vector of errors:

$$e_j=h_j-g\left({\int }_0^{{\displaystyle {\sum }_{i=0}^n}{x}_{ji}}w\left(t\right)dt\right), j=1, 2, \dots , m.$$

Then, if the rank of $J$ is full and the parameter update for $w\left(t\right)$ is performed using the Gauss–Newton optimization method,

$$a^{(k+1)}=a^{\left(k\right)}-{\left(J^{T}J\right)}^{-1}{J}^{T}e,$$

the algorithm is convergent and brings the parameters of the function $w\left(t\right)$ closer to values that minimize $E\left(w\left(t\right)\right)$.

Proof of Theorem 2.

Let the Gauss–Newton method approximate the error function using a quadratic approximation and update parameter $a$ according to the equation

$$a^{(k+1)}=a^{\left(k\right)}-{\left(J^{T}J\right)}^{-1}{J}^{T}e,$$

(51)

where $J$ is the Jacobian matrix containing the partial derivatives of the errors $\left(e_j\right)$ with respect to parameter $a$, and $e$ is the vector of errors for all samples $\left(j\right)$.

The error function of the neuron is represented as

$$E(a)=\sum _{j=1}^m{\left(h_j-g\left({\int }_0^{{\displaystyle {\sum }_{i=0}^n}{x}_{ji}}w\left(t\right)dt \right)\right)}^2,$$

(52)

where $w\left(t\right)$ is the weight function, which depends on parameter $a$. Given that the neuron is at least of the second class ($w\left(t\right)\in C^1$), it follows that the composition $E\left(w\left(t\right)\right)\in C^2$, and it can be quadratically approximated in a small neighborhood of the current solution $a^{\left(k\right)}$ by

$$E\left(a\right)\approx E\left(a^{\left(k\right)}\right)+\nabla {E\left(a^{\left(k\right)}\right)}^T∆a+{\displaystyle \frac{1}{2}}{∆a}^{T}H∆a,$$

(53)

where $\nabla {E\left(a^{\left(k\right)}\right)}^T$ is the transposed gradient of the error function, and $H\approx J^{T}J$ is the Hessian matrix:

$$H={\left[\begin{array}{cc}\begin{array}{c}\begin{array}{ccc}{\displaystyle \frac{{\partial }^{2}E}{{\partial a}_0^2}}& {\displaystyle \frac{{\partial }^{2}E}{\partial a_0\partial a_1}}& \dots \\ {\displaystyle \frac{{\partial }^{2}E}{\partial a_1\partial a_0}}& {\displaystyle \frac{{\partial }^{2}E}{{\partial a}_1^2}}& \dots \\ ⋮& ⋮& \ddots \end{array}\\ \begin{array}{ccc}{\displaystyle \frac{{\partial }^{2}E}{\partial a_q\partial a_0}}& {\displaystyle \frac{{\partial }^{2}E}{\partial a_q\partial a_1}}& \dots \end{array}\end{array}& \begin{array}{c}\begin{array}{c}{\displaystyle \frac{{\partial }^{2}E}{\partial a_0\partial a_q}}\\ {\displaystyle \frac{{\partial }^{2}E}{\partial a_1\partial a_q}}\end{array}\\ ⋮\\ {\displaystyle \frac{{\partial }^{2}E}{{\partial a}_q^2}}\end{array}\end{array}\right]}_{(q+1)\times (q+1)}.$$

(54)

Let

$$∆a=a^{(k+1)}-a^{\left(k\right)}.$$

(55)

From (51) and (55), it follows that

$$∆a=-{\left(J^{T}J\right)}^{-1}{J}^{T}e.$$

(56)

The change in the error function with this update

$$∆E=E\left(a^{(k+1)}\right)-E\left(a^{\left(k\right)}\right)$$

(57)

is represented as

$$∆E=-\nabla {E\left(a^{\left(k\right)}\right)}^T{\left(J^{T}J\right)}^{-1}\nabla E\left(a^{\left(k\right)}\right)+{\displaystyle \frac{1}{2}}\nabla {E\left(a^{\left(k\right)}\right)}^T{\left(J^{T}J\right)}^{-1}H{\left(J^{T}J\right)}^{-1}\nabla E\left(a^{\left(k\right)}\right).$$

(58)

The first term in the change in the error is

$$-\nabla {E\left(a^{\left(k\right)}\right)}^T{\left(J^{T}J\right)}^{-1}\nabla E\left(a^{\left(k\right)}\right).$$

(59)

This term represents the main contribution to the change in $E\left(a\right)$ and is linearly dependent on $∆a$. Since linear dependence is stronger than quadratic dependence for small values of $∆a$ (where the new parameter values $a^{(k+1)}$ remain close to the old $a^{\left(k\right)}$), the first term dominates in the expression for $∆E$.

If the function $E\left(a\right)$ is exactly quadratic, then the second term

$${\displaystyle \frac{1}{2}}\nabla {E\left(a^{\left(k\right)}\right)}^T{\left(J^{T}J\right)}^{-1}H{\left(J^{T}J\right)}^{-1}\nabla E\left(a^{\left(k\right)}\right)$$

(60)

represents the exact contribution of the second derivative to the change $∆E$. In this case, the quadratic approximation is exact, and the approximation error is zero. If the error function is nearly quadratic, this means that the higher derivatives of $E\left(a\right)$ (third-order and above) are very small. This results in a minimal contribution from the second term to the overall change in $E\left(a\right)$. Ultimately, we reach what we set out to prove:

$$∆E<0.$$

(61)

Before we proceed, let us note something important. In the statement of Theorem 2, suitable initial values for the parameters of the weight function were deliberately left undefined. The Gauss–Newton and Levenberg–Marquardt methods (the latter based on the former) use a quadratic approximation of the error function, which performs well only in a small neighborhood around the optimal solution. Therefore, the initial parameters need to be close to the global minimum for the algorithm to converge towards it. However, if the initial parameters are far from the global minimum, the algorithm will tend to find the nearest local minimum. It is important to emphasize that, in all cases, the algorithm will converge to some minimum, which may be local but is not necessarily the global minimum. □

3.4. Algorithm for Finding the Optimal Weight Function Using the Levenberg–Marquardt Method

The Levenberg–Marquardt method combines the gradient descent and Gauss–Newton methods for nonlinear optimization, making it particularly effective for minimization tasks of nonlinear functions, such as the error function in ANNs. In this method, the parameter update for the weight function is represented by the transition (Figure 8)

$$a \leftarrow a-{\left(J^{T}J+\lambda I\right)}^{-1}{J}^{T}e,$$

(62)

where

• $a$ is the vector containing the parameters of the weight function $w\left(t\right)$;
• $J$ is the Jacobian matrix with the partial derivatives of the error $E\left(w\left(t\right)\right)$ with respect to each parameter of $w\left(t\right)$;
• $e$ is the vector containing all errors as each training sample passes through the neuron at each iteration;
• $\lambda$ is the parameter that enables the transition between the gradient descent method and the Gauss–Newton method;
• $I$ is the identity matrix.

Figure 8. Algorithm for finding the optimal weight function using the Levenberg–Marquardt method.

Figure 8. Algorithm for finding the optimal weight function using the Levenberg–Marquardt method.

Theorem 3

(Convergence of the Levenberg–Marquardt Method for Optimizing the Weight Function). Let an integral neuron of at least second class with a smooth transfer function $g$ be given. Let $a$ be the vector containing the parameters of the weight function, $J$ be the Jacobian matrix of the error function $E\left(w\left(t\right)\right)$, and $e$ be the vector of errors

$$e_j=h_j-g\left({\int }_0^{{\displaystyle {\sum }_{i=0}^n}{x}_{ji}}w\left(t\right)dt\right), j=1, 2, \dots , m.$$

Let $\lambda >0$ be the parameter for transitioning between gradient descent $(\lambda \to \infty )$ and the Gauss–Newton method ($\lambda \to 0$), and let $I$ be the identity matrix of the same dimension as $J$.

Then, if the rank of $J$ is full and the parameter update for  $w\left(t\right)$ is performed using the Levenberg–Marquardt optimization method,

$$a^{(k+1)}=a^{\left(k\right)}-{\left(J^{T}J+\lambda I\right)}^{-1}{J}^{T}e,$$

the algorithm is convergent and brings the parameters of the function $w\left(t\right)$ closer to values that minimize  $E\left(w\left(t\right)\right)$.

Proof of Theorem 3.

The proof of this theorem is similar to that of Theorem 2, concerning the Gauss–Newton method. We again consider a neuron of at least second class with a smooth transfer function $g$, which is trained with examples from the set ${D=\left\{[x_{j0}=1,x_{j1},\dots ,x_{jn}; h_j]\right\}}_{j=1}^m$.

Since the MSE function is represented as

$$E\left(w\left(t\right)\right)=\sum _{j=1}^m{\left(h_j-g\left({\int }_0^{{\displaystyle {\sum }_{i=0}^n}{x}_{ji}}w\left(t\right)dt \right)\right)}^2,$$

(63)

and given that the neuron is at least of the second class, it follows that the composition $E\left(w\left(t\right)\right)\in C^2$. To show that the algorithm

$$a \leftarrow a-{\left(J^{T}J+\lambda I\right)}^{-1}{J}^{T}e$$

(64)

is convergent, we need to show that

$$E\left(a^{\left(k+1\right)}\right)<E\left(a^{\left(k+1\right)}\right).$$

(65)

Formally, the parameter update $a$ at the $(k+1)$-th iteration is expressed by the equation

$$a^{(k+1)}=a^{\left(k\right)}-{\left(J^{T}J+\lambda I\right)}^{-1}{J}^{T}e.$$

(66)

Let $∆a$ denote the matrix $\left(a^{(k+1)}-a^{\left(k\right)}\right)$. It is evident that

$$∆a=-{\left(J^{T}J+\lambda I\right)}^{-1}{J}^{T}e.$$

(67)

The error function $E\left(a\right)$ can be simplified in an infinitesimally small neighborhood of the current value $a^{\left(k\right)}$ through a quadratic approximation:

$$E(a)\approx E\left(a^{\left(k\right)}\right)+\nabla {E\left(a^{\left(k\right)}\right)}^T∆a+{\displaystyle \frac{1}{2}}{∆a}^{T}H∆a,$$

(68)

where

• $\nabla {E\left(a^{\left(k\right)}\right)}^T$ is the transposed gradient of the error function;
• $H\approx J^{T}J$ is the Hessian matrix.

Then, if we substitute the change (67) in the parameter update for the weight function $w\left(t\right)$ into expression (68) and denote

$$∆E=E\left(a^{(k+1)}\right)-E\left(a^{\left(k\right)}\right)$$

(69)

we obtain

$$∆E=-\nabla {E\left(a^{\left(k\right)}\right)}^T{\left(J^{T}J+\lambda I\right)}^{-1}\nabla E\left(a^{\left(k\right)}\right)+{\displaystyle \frac{1}{2}}\nabla {E\left(a^{\left(k\right)}\right)}^T{\left(J^{T}J+\lambda I\right)}^{-1}H{\left(J^{T}J+\lambda I\right)}^{-1}\nabla E\left(a^{\left(k\right)}\right).$$

(70)

Since $\lambda >0$ and $(J^{T}J+\lambda I)$ is positive definite, we can maintain the Levenberg–Marquardt optimization idea of dynamically adjusting $\lambda$ during training:

• If $∆E<0$, then the update is successful, and the error has decreased. In this case, the algorithm decreases $\lambda$, allowing for larger steps in the next iteration.
• If $∆E\ge 0$, then the update is unsuccessful, and the error has increased. In this case, $\lambda$ is increased, which limits the step size and makes the behavior closer to gradient descent.

Thus, ultimately, in each iteration, the Levenberg–Marquardt algorithm will adjust the parameters of the weight function to reduce the error function $E$. Since the error function is at least twice differentiable and the algorithm dynamically adapts the step size $\lambda$, convergence to a minimum is guaranteed, even if we start with arbitrary initial parameter values. □

3.5. Selection of Weight Function and Algorithm Complexity

The determination of an appropriate weight function $w\left(t\right)$ depends on the specifics of the problem itself. Polynomial, exponential, and trigonometric functions are often a good choice due to their flexibility and ease of parameterization. For problems requiring interpolation or subsequent extrapolation, polynomial weight functions are an excellent choice. However, if the goal is to use the integral neuron to describe decaying dependencies, exponential weight functions would be more suitable. Gaussian weight functions are appropriate for processing local dependencies, and so on. Moreover, considering the fact that the weight function essentially determines the signal within the neuron’s body, which is then passed as an argument to the transfer function, several details should be noted that might prove critical when selecting the weight function:

1.

Interval and values of input data:

The weight function must be compatible with the distribution of input values. If the inputs are defined within a specific range, $w\left(t\right)$ can be as follows:

-

A polynomial function for uniform distribution;

-

A Gaussian function for local dependencies, where $w\left(t\right)$ can be adjusted to a specific sub-range through parameters related to the mean and standard deviation (variance).

2.

Transfer function $g\left(S\right):$

The selection of $w(t$) should also take into account the characteristics of the transfer function:

-

For transfer functions such as $tanh\left(S\right)$ or the sigmoid transfer function $\sigma \left(S\right),w\left(t\right)$ should ensure that the signal $S$ falls within regions where the transfer function is most sensitive (e.g., around $S=0$). For instance, polynomial and exponential weight functions can be scaled to direct the signal $S$ to optimal ranges.

-

For transfer functions with boundaries or whose domain excludes certain points, the weight function must constrain the signal $S$ to a range compatible with the required restrictions. For example, for the reverse sigmoid,

$$g\left(S\right)={\displaystyle \frac{1}{1+e^{-\frac{1}{s}}}},$$

or for the modified $tanh\left(S\right)$,

$$g\left(S\right)=\tanh \left(S\right)+{\displaystyle \frac{1}{S^2}}$$

the weight function $w\left(t\right)$, which forms the signal

$$S={\int }_0^{{\displaystyle {\sum }_{i=0}^n}{x}_i}w\left(t\right)dt,$$

must be chosen to prevent situations where $S=0$, as this would render the use of transfer functions like $tanh\left(S\right)$ and $\sigma \left(S\right)$ impossible.

3.

Smoothness and computational complexity:

The weight function should be at least $C^k$ (smooth up to the $k-th order$), to facilitate the following:

-

The calculation of gradients and the Jacobian and Hessian matrix during training;

-

The correct behavior of the integral.

From the perspective of computational complexity, different choices of weight functions can be considered. Polynomial functions are easy to compute and differentiate, while more complex weight functions (e.g., $w\left(t\right)=sin\left(t\right)$ or $w\left(t\right)=e^{-t^2}$) offer additional flexibility but are computationally more expensive.

4.

Nature of the problem:

The type of task (regression, classification, modeling dependencies) influences the choice:

-

Regression: Polynomial or exponential weight functions provide smooth approximation.

-

Classification: Locally sensitive weight functions (such as Gaussian) can aid in class separation.

-

Temporal tasks: Exponential weight functions can model a decaying effect over time.

Ultimately, the problem of selecting the most appropriate weight function for the integral neuron is the same as the problem of choosing a transfer function for standard neurons. In the theory of artificial neural networks, there is still no objective rule for this. Of course, it is advisable for the weight function to be continuous, monotonic, and differentiable, but which specific function would be the most useful remains a matter of additional judgment. Based on their experience, researchers develop professional intuition that helps them determine what type of transfer function is needed under specific conditions defined by the task and the other parameters of the neurons being used. The exact, specific type is determined as a result of experimentation and comparison of results. For this purpose, algorithms are even being developed for automated search for suitable neurons. Similarly, the same principles apply to the optimal selection of the weight function.

4. Discussion: Classical and Integral Neurons and Neural Networks

It turns out that the optimization algorithms—gradient descent, Gauss–Newton, and their generalization, the Levenberg–Marquardt method—are convergent and, starting from random parameters for the weight function, can optimize it in a way that indeed minimizes the error. Here, it is essential to emphasize something critical. The condition for the proximity of input data that we imposed from the start, $\lambda =\max \left\{∆x_1, ∆x_2, \dots ,∆x_n\right\}\to 0$ for $∆x_i=x_i-x_{i-1}, i=1,2,\dots , n$, ensured the possibility of representing

$$S=\sum _{i=0}^n{w}_i{x}_i={\int }_0^{{\displaystyle {\sum }_{i=0}^n}{x}_i}w\left(t\right)dt.$$

(71)

This condition was solely driven by our desire to establish equivalence between classical neurons and those with weight functions. However, the initial conditions of the approximation problem are often not like this. In the general case, the input data ${\{x_{j0},x_{j1},\dots ,x_{jn}\}}_{j=1 }^m$ provided for each sample can have values spaced far apart ($\lambda$ has a large value), which could introduce error in the representation (71). In these cases, a commonly used approach is data normalization. If the input–output data are transformed via appropriate normalization to fit within a small interval, (71) will hold with negligible error, giving us a weight function that returns different weights along the dendrites depending on the dataset. This effectively endows the neuron with dynamic characteristics.

In the opposite case, if the requirement $\lambda =\max \left\{∆x_1, ∆x_2, \dots ,∆x_n\right\}\to 0$ is disregarded, entirely new neural structures emerge that have no standard equivalent. However, the cumulative signal within their bodies is still obtained from the value of the integral in (71) with some arbitrary continuous and smooth function $w\left(t\right)$:

$$\tilde{S}={\int }_0^{{\displaystyle {\sum }_{i=0}^n}{x}_i}w\left(t\right)dt.$$

(72)

In this case, Theorems 1–3 will still hold, which is very important. By abandoning the requirement $\lambda =\max \left\{∆x_1, ∆x_2, \dots ,∆x_n\right\}\to 0$, we see the possibility of existing and training neural structures characterized by the absence of a true dendritic tree, where each input dataset is provided as a set through only one dendritic segment with a weight of 1 (Figure 9).

Since, for these integral neurons, it is not required that $\lambda =\max \left\{∆x_1, ∆x_2, \dots ,∆x_n\right\}\to 0$, it follows that we cannot assign them a classical equivalent with weights $w_i$ for which

$$\sum _{i=0}^n{w}_i{x}_i={\int }_0^{{\displaystyle {\sum }_{i=0}^n}{x}_i}w\left(t\right)dt.$$

(73)

This makes integral neurons an independent class, distinct from the classical structures we are accustomed to working with. Considering that

$$\sum _{i=0}^n{x}_i=\sum _{i=0}^n{w_{i}x}_i, \forall w_i=1,$$

(74)

it follows that we can represent the integral neuron using familiar schemes where the dendritic branches have the same transmissive strength, equal to 1 (Figure 10).

The conditions in Theorems 1–3 do not require the proximity of input data, $\lambda =\max \left\{∆x_1, ∆x_2, \dots ,∆x_n\right\}\to 0$, making them applicable to integral neurons as well. With these theorems, we have theoretically demonstrated that, just like classical neural structures, integral neural structures can be trained with samples and ultimately approximate a given problem with minimal error.

In classical neurons, training algorithms focus on modifying the weights, which influence the signal within the neuron body and, subsequently, the output. In integral neural structures, the weights of input data are fixed and constant with a value of 1, and the training algorithms are directed specifically at the parameters of the weight function—that is, the optimization of the type and shape of the signal within the neurons (Figure 11).

Another avenue for discussion is the possibility of creating neural structures that contain both integral and classical neurons simultaneously. Although this study examines only individual integral neurons, definitions can be proposed for the new integral structures created with their assistance.

Definition 3

(Integral Neural Network). An artificial neural network consisting entirely of integral neurons of class $(k+1)$ will be called a fully integral neural network of class $\left(k+1\right),k\in N_0$.

Definition 4

(Network with Integral–Classical Architecture). A neural network containing $i$ layers of integral neurons of class  $(k+1)$ and $t$ layers of classical neurons with a linear cumulative signal in the body $(i , t\in N)$ will be called a neural network with integral–classical architecture of class $\left(k+1\right)$ and signature $(i, t)$.

A neural network with an integral–classical architecture combines layers of integral and classical neurons organized in a sequential or parallel structure, enabling both the processing of input stimuli through integral weight functions and calculations via linear combinations of stimuli, depending on the specific task and training requirements. In this type of hybrid architecture, integral layers can be seen with neurons, in the bodies of which we have a signal,

$$\tilde{S}={\int }_0^{{\displaystyle {\sum }_{i\in I}}{x}_i}w\left(t\right)dt,$$

(75)

as well as classical structures with a linear signal in the bodies

$$\tilde{S}={\int }_0^{{\displaystyle {\sum }_{i\in I}}{x}_i}w\left(t\right)dt,$$

(76)

where $I$ and $J$ are the sets of indices of neurons from previous layers that provide output signals to the current layer. This hybrid architecture combines the advantages of both neuron types, allowing flexible (integral and nonlinear) representation of dependencies among inputs and standard processing through classical linear sums. It is evident that a fully integral neural network with $i$ hidden layers can be considered and exhibits all the characteristics of an integral–classical network with signature $(i,0)$, and any classical neural network with $j$ hidden layers can be regarded as an integral–classical architecture with signature $\left(0,j\right)$, where $i,j\in N$.

Definition 5

(Architectural Balance). Let $Net$ be a neural network with an integral–classical architecture of signature $(i, t)$. The ratio

$$p_{Net}={\displaystyle \frac{{\displaystyle {\sum }_{k=1}^i}{N}_k}{{\sum }_{m=1}^t{N}_m}}$$

(77)

where $N_k$ and $N_m$ are the numbers of integral and classical neurons in the respective layers will be called the architectural balance coefficient. We will say that the neural network $Net$ has an ideal architectural balance if and only if $i=t$ and $p_{Net}=1$. Such a neural network will be referred to as integrally and classically balanced.

By adjusting the signature ($i,t$), thereby altering the architectural balance of the neural structure, we can optimize the network for specific tasks. For tasks with higher demands for nonlinear representation, the number of integral layers and neurons can be increased, while for tasks with simpler dependencies, a greater emphasis can be placed on classical layers and neurons. Naturally, it should be acknowledged that, although an integral–classical architecture combines the strengths of both approaches—the efficiency of linear computations and the power of integral dependencies—training this type of structure presents a true scientific and practical challenge. One of the team’s goals is to develop multitask training where integral and classical layers are trained with partially independent algorithms, but with coordinated optimization steps across iterations.

5. Integral XOR Neuron

A particularly important topic in the theory of artificial neural networks relates to the XOR logical function and the problem that arises from the inability to train a single neuron to solve XOR.

For the logical functions “AND” and “OR”, it is straightforward to construct a linear classifier and, therefore, a single neuron that recognizes objects from the two classes of points, returning “1” or “0”, respectively (Figure 12).

The recognition of different classes is due to the fact that a signal is formed within the body of a standard neuron:

$$S=w_1{x}_1+w_2{x}_2+b,$$

(78)

which, for $S=0$, is equivalent to the line

$$w_1{x}_1+w_2{x}_2+b=0$$

(79)

that separates the classes. For points on one side of the line, $S>0$, and for those on the other side, $S<0$. The division of points into two classes is achieved using an appropriate (typically step) transfer function.

On the other hand, “exclusive OR” (XOR) is a logical operation between two Boolean variables that returns true (“1”) if and only if the inputs are different. The XOR truth

Table 1. The XOR truth table.

${\mathit{X}}_{\mathbf{1}}$	${\mathit{X}}_{\mathbf{2}}$	${\mathit{X}}_{\mathbf{1}}\mathit{X}\mathit{O}\mathit{R}{\mathit{X}}_{\mathbf{2}}$
0	0	0
0	1	1
1	0	1
1	1	0

is as follows:

The key problem with XOR and a single neuron lies in the fact that XOR cannot be solved through linear classification, as there is no single line that can separate the input values (0,0) and (1,1), which return “0,” from (0,1) and (1,0), which return “1”, in a two-dimensional space (Figure 13). This means that a perceptron cannot find weights that successfully separate the ordered input pairs $(X_1, X_2)$ based on their XOR output values.

This problem was highlighted as early as 1969 in the book by Marvin Minsky and Seymour Papert [67], which led to a temporary decline in interest in neural network research. The authors demonstrated that single-layer perceptrons are limited and cannot be trained on certain types of logical functions, such as XOR.

To overcome the limitation of a single perceptron, multilayer perceptrons (MLPs) were introduced, which include one or more hidden layers. These layers enable the ANN to learn and represent nonlinear relationships, like XOR, by using combinations of linear classifications connected through nonlinear transfer functions, such as sigmoid or ReLU. Thus, MLPs can create complex, nonlinear separating surfaces that successfully model and solve XOR and similar functions. However, the problem of representing XOR with a single neuron remains unresolved in the theory of ANNs. One reason for this is that, since the advent of artificial neurons, the signal formed within their body has been conceptualized in the same way.

What would happen if the sum in the neuron’s body lost its linear, discrete nature,

$$S=\sum _{i=0}^n{w}_i{x}_i,$$

(80)

and was instead replaced by

$$\tilde{S}={\int }_0^{{\displaystyle {\sum }_{i=0}^n}{w_{i}x}_i}w\left(t\right)dt, \forall { w}_i=1$$

(81)

where $w\left(t\right)$ is the neuron’s weight function? In the case of the XOR logical function, we consider a neuron with two inputs (Figure 14), which we will refer to as an integral XOR neuron.

The set of training samples for the neuron is represented as

$${D=\left\{\left[(x_{j1},x_{j2}); h_j\right]\right\}}_{j=1}^4,$$

(82)

where the input–output data values are as specified in

Table 2. Input-output data for training the XOR neuron.

j	${\mathit{X}}_{\mathit{j}1}$	${\mathit{X}}_{\mathit{j}\mathbf{2}}$	${\mathit{h}}_{\mathit{j}}={\mathit{X}}_{\mathit{j}\mathbf{1}}\mathit{X}\mathit{O}\mathit{R}{\mathit{X}}_{\mathit{j}\mathbf{2}}$
1	0	0	0
2	0	1	1
3	1	0	1
4	1	1	0

.

Before presenting the results of our theoretical considerations and their experimental confirmation, it is important to note that the Levenberg–Marquardt algorithm is sensitive to errors and to the training stage. With a higher amplitude transfer function, neuron output errors are detected and corrected more effectively, leading to a faster convergence to the error function minimum and improved overall convergence. Many transfer functions, including the hyperbolic tangent ($tanh$), have relatively smooth gradients around zero, which can slow down training.

On the other hand, scaling transfer functions by coefficients does not alter their shape but intensifies their effect, increasing gradients and amplitudes, which makes training more efficient. Therefore, we have chosen the transfer function $g(\tilde{S})=3\mathrm{t}\mathrm{a}\mathrm{n}\mathrm{h}(\tilde{S})$. We deliberately multiply by three to increase the neuron’s output amplitude, meaning that the output signal becomes stronger for the same input values. This allows the neuron to cover a broader range of values and respond more strongly to changes in its inputs, helping the training process capture and correct errors more effectively.

Subsequent experiments with linear and sinusoidal transfer functions also demonstrated their viability in building an integral XOR neuron. In all three experiments, a first-degree polynomial $w\left(t\right)=a_0+a_{1}t$ was used as the weight function.

In artificial neural network theory, there is no strict rule or objective criterion for selecting an appropriate transfer function. Researchers typically choose a transfer function based on prior experience and professional intuition, considering the specific task or problem requirements. For instance, continuous transfer functions with sigmoid characteristics are known to solve a wide range of problems, but the specific choice of function involves experimentation combined with past experience.

Similarly, the choice of weight function follows the same principle. While certain criteria—such as continuity and smoothness—are essential, the exact form of the weight function depends on the particular task. Considering the XOR problem, with only four training samples (Table 1), it is appropriate to select a polynomial weight function with a maximum degree of $k=3$, possessing four parameters. However, due to the characteristics of the transfer functions used, experimentation revealed that the optimal weight function is based on a first-degree polynomial $w\left(t\right)=a_0+a_{1}t$, characterized by two parameters ($a_0,{ a}_1$), whose optimal values are determined by the training algorithms.

The MATLAB v2018a scripts for creating, training, and operating integral XOR neurons are provided in Appendix A of this article.

5.1. Solving XOR with an Integral Neuron Using a Hyperbolic Tangent Transfer Function

5.1.1. Theoretical Solution

For the weight function, we choose a first-degree polynomial:

$$w\left(t\right)=a_0+a_{1}t,$$

(83)

where $a=(a_0, a_1)$ are the parameters of $w\left(t\right)$ that need to be optimized.

The error function is represented as

$$E\left(a\right)=\sum _{j=1}^4{\left(h_j-3\tanh \left( \tilde{S} \right)\right)}^2,$$

(84)

where

$$\tilde{S}={\int }_0^{{\displaystyle {\sum }_{i=1}^2}{w_{i}x}_i}w\left(t\right)dt=a_0\left(w_1{x}_1+w_2{x}_2\right)+{\displaystyle \frac{1}{2}}{a}_1{\left(w_1{x}_1+w_2{x}_2\right)}^2, w_1=w_2=1.$$

(85)

We will prove that parameters ${a=(a}_0,a_1)$ for the weight function $w\left(t\right)$ exist, such that the neuron can solve the XOR logical function. In practice, we need to show that ∃${ a=(a}_0,a_1)$ such that

$$\left\{\begin{array}{c}Out\left(0,0\right)=3\tanh \left(\tilde{S}\left(0,0\right)\right)=0\\ Out\left(0,1\right)=3\tanh \left(\tilde{S}\left(0,1\right)\right)=1\\ Out\left(1,0\right)=3\tanh \left(\tilde{S}\left(1,0\right)\right)=1\\ Out\left(1,1\right)=3\tanh \left(\tilde{S}\left(1,1\right)\right)=0\end{array}\right..$$

(86)

The first equality

$$Out\left(0,0\right)=3\tanh \left(\tilde{S}\left(0,0\right)\right)=3\mathrm{t}\mathrm{a}\mathrm{n}\mathrm{h}\left(a_0\left(0+0\right)+{\displaystyle \frac{1}{2}}{a}_1{\left(0+0\right)}^2\right)=3\mathrm{t}\mathrm{a}\mathrm{n}\mathrm{h}(0)=0$$

(87)

holds for any ${a=(a}_0,a_1)\in R$.

The second and third equalities in (86), due to the nature of $\tilde{S}$, provide the same information,

$$3tanh\left(a_0+{\displaystyle \frac{1}{2}}{a}_1\right)=1,$$

(88)

which implies

$$a_0+{\displaystyle \frac{1}{2}}{a}_1={\displaystyle \frac{1}{2}}\ln \left(2\right).$$

(89)

Finally, from the last equality in (86), we have

$$3\mathrm{t}\mathrm{a}\mathrm{n}\mathrm{h}\left({2(a}_0+a_1)\right)=0,$$

(90)

i.e.,

$$a_1=-a_0.$$

(91)

Then, from (89) and (91), we can form the system

$$\left\{\begin{array}{l}{a}_0+{\displaystyle \frac{1}{2}}{a}_1={\displaystyle \frac{1}{2}}\mathrm{l}\mathrm{n}(2)\\ a_1={-a}_0\end{array}\right..$$

(92)

From this, it follows that $\left(a_0,a_1\right)=\left(\ln \left(2\right),-\ln \left(2\right)\right)$.

Thus, in the body of the XOR neuron, we have the signal

$$\tilde{S}=\ln \left(2\right)\left(w_1{x}_1+w_2{x}_2\right)-{\displaystyle \frac{1}{2}}\ln \left(2\right){\left(w_1{x}_1+w_2{x}_2\right)}^2,$$

(93)

and the neuron itself appears as shown in Figure 15.

This integral neuron should be able to solve the XOR logical function.

5.1.2. Experimental Confirmation

For the purposes of the experiments, a MATLAB v2018a script was created (see Appendix B) that implements our proposed Levenberg–Marquardt algorithm for training an integral neuron. When using this script, arbitrary initial values for the vector $a=(a_0, a_1)$, containing the parameters of the weight function, are set and then updated during training using the transition

$$a \leftarrow a-{\left(J^{T}J+\lambda I\right)}^{-1}{J}^{T}e,$$

(94)

where $J$ is the Jacobian matrix containing the derivatives of the error $E\left(a\right)$ with respect to each parameter of $w\left(t\right)$, $e$ is the vector containing all errors for each sample in the training set that passes through the neuron at each iteration, $\lambda$ is the parameter that controls the transition between the gradient descent method and the Gauss–Newton algorithm, and $I$ is the identity matrix.

During the iterations, the following parameters are tracked:

• Iteration: This is the number of the current iteration of the algorithm. The algorithm goes through multiple iterations, updating parameters each time in an attempt to find the optimal solution. Iterations continue until the maximum number of iterations is reached or convergence criteria are met (e.g., when changes in error or step size become very small).
• Func-count: This is the number of calls to the objective function, which calculates the error (residuals). Monitoring the number of calls is essential because each call requires computations that can be costly for complex tasks. The algorithm may call the function more than once per iteration to calculate gradients or other necessary quantities.
• First-Order Optimality: This measures how close the current solution is to the optimum, calculated using the gradient of the error function. In optimization, first-order optimality measures how far the gradient is from zero. A gradient close to zero suggests proximity to the optimal solution. If the first-order optimality value is low, it is an indication that the algorithm may be near a local (or global) minimum.
• Lambda ($\lambda$): This is the Levenberg–Marquardt parameter controlling the transition between the gradient descent method and the Gauss–Newton method. At high values of $\lambda$, the algorithm behaves like gradient descent, taking smaller, more conservative steps. When $\lambda$ is low, the algorithm approaches the Gauss–Newton method, making more aggressive steps that can lead to faster convergence. The algorithm dynamically adjusts $\lambda$ based on the success of previous iterations.
• Norm of Step: The step norm indicates the magnitude of the parameter change in the current iteration. Large step norm values usually mean the algorithm is making significant adjustments to parameters, while small values indicate the algorithm is fine-tuning and making more minor adjustments. When the step norm becomes very small, it usually signifies that the algorithm is close to convergence.

To start the algorithm, the matrix $a$ is randomly initialized with values in the range [−1,1]. For the results presented in

Table 3. Results of the Levenberg–Marquardt method for finding the optimal weight function of the XOR neuron.

Iteration	Func-Count	Residual	First-Order Optimality	Lambda	Norm of Step
0	3	12.4149	3.39	0.01
1	7	7.73309	15	0.1	1.41445
2	10	1.03079	5.34	0.01	0.423156
3	13	0.007648	0.536	0.001	0.445144
4	16	1.88 × 10−9	0.000306	0.0001	0.030293
5	19	9.98 × 10−21	3.08 × 10−10	1.00 × 10−5	4.85 × 10−6
6	22	5.42 × 10−31	3.40 × 10−15	1.00 × 10−6	7.84 × 10−11
7	25	0	0	1.00 × 10−7	4.78 × 10−16

, the algorithm randomly selected initial values for the weight function parameters as follows: $a_0=0.629447372786358, a_1=0.811583874151238$.

As is typical in artificial intelligence, initially, with the randomly selected parameters of its weight function $w\left(t\right)$, the integral neuron is not adapted to the problem it needs to solve, resulting in an error of 12.4149. However, as the iterations proceed, the parameters of $w\left(t\right)$ gradually change, leading to increasing effectiveness until reaching zero error. In this case, the algorithm identifies the optimal parameters for the weight function as $a(a_0=0.693147180559945, a_1=-0.693147180559945)$, making the final form of the weight function

$$w\left(t\right)=0.693147180559945-0.693147180559945t.$$

(95)

It is noteworthy that $0.693147180559945=ln\left(2\right)$, as predicted by theory, indicating that the Levenberg–Marquardt algorithm has accurately reached the global minimum of the error function.

The results returned by the integral neuron with this weight function are presented in

Table 4. Results of the integral neuron with initial parameters of the weight function and hyperbolic tangent transfer function.

Inputs $\mathbf{(}{\mathit{x}}_{\mathbf{1}}\mathbf{,}{\mathit{x}}_{\mathbf{2}}\mathbf{)}$	Neuron Body $\stackrel{\mathbf{~}}{\mathit{S}}\mathbf{=}{\int }_{\mathbf{0}}^{{\mathit{x}}_{\mathbf{1}}\mathbf{+}{\mathit{x}}_{\mathbf{2}}}\mathit{w}\left(\mathit{t}\right)\mathit{d}\mathit{t}$	Neuron Output $\mathit{O}\mathit{u}\mathit{t}\mathbf{=}\mathbf{3}\mathbf{tanh}(\stackrel{\mathbf{~}}{\mathit{S}})$
(0,0)	0	0
(0,1)	0.346573590279973	1
(1,0)	0.346573590279973	1
(1,1)	0	0

.

5.2. Solving XOR with an Integral Neuron Using the Transfer Function $f\left(x\right)=x$

5.2.1. Theoretical Solution

By analogy with the theoretical considerations we made for an integral neuron with the transfer function $3tanh$ and with the same choice of weight function, we conclude that the optimal parameters satisfying the equations

$$\left\{\begin{array}{c}Out\left(0,0\right)=\left[\tilde{S}\left(0,0\right)\right]=0\\ Out\left(0,1\right)=\left[\tilde{S}\left(0,1\right)\right]=1\\ Out\left(1,0\right)=\left[\tilde{S}\left(1,0\right)\right]=1\\ Out\left(1,1\right)=\left[\tilde{S}\left(1,1\right)\right]=0\end{array}\right.$$

(96)

are solutions to the system

$$\left\{\begin{array}{l}{a}_0+{\displaystyle \frac{1}{2}}{a}_1=1\\ a_1={-a}_0\end{array}\right..$$

(97)

The solution to (97) leads to optimal parameters for the weight function of ${(a}_0=2, a_1=-2)$, which means that the integral neuron capable of handling XOR would appear as shown in Figure 16.

5.2.2. Experimental Confirmation

Applying the Levenberg–Marquardt method to the parameters of the weight function, with an initial random selection of $a_0=-0.746026367412988, a_1=0.826751712278039$, the algorithm, through its iterations, reached the pair ${(a}_0=2, a_1=-2)$, as predicted by theory, defining the global minimum of the error (

Table 5. Results of the Levenberg–Marquardt method for finding the optimal weight function of the XOR neuron with linear transfer function $f\left(x\right)=x$.

Iteration	Func-Count	Residual	First-Order Optimality	Lambda	Norm of Step
0	3	3.57798	2.34	0.01
1	6	0.007213	0.0268	0.001	3.74853
2	9	1.90 × 10−7	0.000145	0.0001	0.19179
3	12	5.03 × 10−14	7.48 × 10−8	1.00 × 10−5	0.000988
4	15	1.33 × 10−22	3.85 × 10−12	1.00 × 10−6	5.09 × 10−7
5	18	0	0	1.00 × 10−7	2.62 × 10−11

).

5.3. Solving XOR with an Integral Neuron Using the Transfer Function $f\left(x\right)=2sin\left(x\right)$

5.3.1. Theoretical Solution

Let us consider a neuron with the transfer function $g\left(x\right)=2sin\left(x\right)$, where the weight function is again a first-degree polynomial:

$$w\left(t\right)=a_0+a_{1}t.$$

(98)

The signal in the body of the neuron is

$$\tilde{S}={\int }_0^{{\displaystyle {\sum }_{i=1}^2}{w_{i}x}_i}w\left(t\right)dt=a_0\left(w_1{x}_1+w_2{x}_2\right)+{\displaystyle \frac{1}{2}}{a}_1{\left(w_1{x}_1+w_2{x}_2\right)}^2, w_1=w_2=1.$$

(99)

Given that $w_1=w_2=1$, it is clear that the neuron’s output is

$$Out=2\mathrm{s}\mathrm{i}\mathrm{n}\left(a_0\left(x_1+x_2\right)+{\displaystyle \frac{1}{2}}{a}_1{\left(x_1+x_2\right)}^2\right).$$

(100)

We will prove that parameters ${a=(a}_0,a_1)$ exist for the weight function $w\left(t\right)$ that enable the neuron to solve the XOR logical function. In practice, we need to show that ∃${ a=(a}_0,a_1)$ such that

$$\left\{\begin{array}{c}Out\left(0,0\right)=2\sin \left(\tilde{S}\left(0,0\right)\right)=0\\ Out\left(0,1\right)=2\sin \left(\tilde{S}\left(0,1\right)\right)=1\\ Out\left(1,0\right)=2\sin \left(\tilde{S}\left(1,0\right)\right)=1\\ Out\left(1,1\right)=2\sin \left(\tilde{S}\left(1,1\right)\right)=0\end{array}\right..$$

(101)

The first equality

$$Out\left(0,0\right)=2\sin \left(\tilde{S}\left(0,0\right)\right)=2\mathrm{s}\mathrm{i}\mathrm{n}\left(a_0\left(0+0\right)+{\displaystyle \frac{1}{2}}{a}_1{\left(0+0\right)}^2\right)=2\sin \left(0\right)=0$$

(102)

is satisfied for all ${a=(a}_0,a_1)\in R$.

The second and third equalities in (101), due to the nature of $\tilde{S}$, provide the same information:

$$2\mathrm{s}\mathrm{i}\mathrm{n}\left(a_0+{\displaystyle \frac{1}{2}}{a}_1\right)=1,$$

(103)

which implies

$$a_0+{\displaystyle \frac{1}{2}}{a}_1={\displaystyle \frac{\pi }{6}}+2k\pi , kϵZ$$

(104)

or

$$a_0+{\displaystyle \frac{1}{2}}{a}_1={\displaystyle \frac{5\pi }{6}}+2k\pi , kϵZ.$$

(105)

Finally, from the last equality in (101), we have

$$2\mathrm{s}\mathrm{i}\mathrm{n}\left({2(a}_0+a_1)\right)=0,$$

(106)

i.e.,

$$a_1=-a_0$$

(107)

Let us consider one of the possible systems:

$$\left\{\begin{array}{l}{a}_0+{\displaystyle \frac{1}{2}}{a}_1={\displaystyle \frac{\pi }{6}}+2k\pi \\ a_1={-a}_0\end{array}, kϵZ.\right.$$

(108)

The system has infinitely many solutions, and one solution for $k=0$, $\left({ a}_0, { a}_1\right)=({\displaystyle \frac{\pi }{3}},-{\displaystyle \frac{\pi }{3}})$. With these parameters, the neuron has a weight function

$$w\left(t\right)={\displaystyle \frac{\pi }{3}}\left(1-t\right),$$

(109)

and in its body, we obtain the cumulative signal

$$\tilde{S}={\int }_0^{{\displaystyle {\sum }_{i=1}^2}{x}_i}w\left(t\right)dt={\displaystyle \frac{\pi }{3}}\left(x_1+x_2\right)-{\displaystyle \frac{\pi }{6}}{\left(x_1+x_2\right)}^2.$$

(110)

Along the neuron’s axon (Figure 17), the output signal propagates as

$$Out=2\mathrm{s}\mathrm{i}\mathrm{n}\left({\displaystyle \frac{\pi }{3}}\left(x_1+x_2\right)-{\displaystyle \frac{\pi }{6}}{\left(x_1+x_2\right)}^2\right).$$

(111)

It can easily be seen that, theoretically, this neuron can indeed solve the XOR logical function problem (

Table 6. Theoretical solution of the XOR logical function with an integral neuron using the transfer function $g\left(x\right)=2sin\left(x\right)$.

Inputs $\mathbf{(}{\mathit{x}}_{\mathbf{1}}\mathbf{,}{\mathit{x}}_{\mathbf{2}}\mathbf{)}$	Neuron Body $\stackrel{\mathbf{~}}{\mathit{S}}\mathbf{=}\frac{\mathit{\pi }}{\mathbf{3}}\left({\mathit{x}}_{\mathbf{1}}\mathbf{+}{\mathit{x}}_{\mathbf{2}}\right)\mathbf{-}\frac{\mathit{\pi }}{\mathbf{6}}{\left({\mathit{x}}_{\mathbf{1}}\mathbf{+}{\mathit{x}}_{\mathbf{2}}\right)}^{\mathbf{2}}$	Neuron Output $\mathit{O}\mathit{u}\mathit{t}=3\mathbf{tanh}(\tilde{\mathit{S}})$
(0,0)	0	0
(0,1)	$\pi /6$	1
(1,0)	$\pi /6$	1
(1,1)	0	0

).

5.3.2. Experimental Confirmation

The empirical confirmation of the existence and trainability of this type of neuron was again carried out in the MATLAB v2018a environment. The transfer function $g\left(x\right)=2sin\left(x\right)$, like those previously used, is continuous and smooth, fulfilling the conditions for the applicability of the Levenberg–Marquardt method according to Theorem 3. The initial conditions (initial parameters of the weight function) are once again generated as random numbers within the segment [−1,1]. Given this random initialization, it is expected to obtain various final results for the optimal parameters of the weight function. In the worst observed case, the neuron successfully handles the XOR function with an error on the order of ${10}^{-29}$. In many instances, the algorithm reaches the global minimum, discovering the weight function for a neuron that operates with zero error.

Table 7. Results from the application of the Levenberg–Marquardt Method for finding the optimal weight function of the XOR neuron equipped with the transfer function $g\left(x\right)=2sin\left(x\right)$.

Iteration	Func-Count	Residual	First-Order Optimality	Lambda	Norm of Step
0	3	4.80645	7.29	0.01
1	6	0.000443	0.0404	0.001	2.02773
2	9	3.22 × 10−9	8.10 × 10−5	0.0001	0.025227
3	12	8.72 × 10−17	5.72 × 10−9	1.00 × 10−5	7.08 × 10−5
4	15	2.28 × 10−26	8.81 × 10−14	1.00 × 10−6	1.19 × 10−8
5	18	2.47 × 10−32	3.85 × 10−16	1.00 × 10−7	1.92 × 10−13
6	21	0	0	1.00 × 10−8	1.81 × 10−16

presents an example where the iterative algorithm completes in just six epochs, resulting in a final zero total error.

With initial, random parameters of the weight function $\left(a_0=-0.447949846002843, a_1=0.359405353707350\right)$, before training, the integral neuron produced an error of $4.80645$ when solving XOR. After the final iteration of the training algorithm, it adjusted the parameters to $\left(a_0=1.047197551196598, a_1=-1.047197551196598\right)$, yielding an absolutely accurate result for XOR (

Table 8. Experimental results of the neuron’s performance with final parameters of the weight function and transfer function $g\left(x\right)=2sin\left(x\right)$.

Inputs $\mathbf{(}{\mathit{x}}_{\mathbf{1}}\mathbf{,}{\mathit{x}}_{\mathbf{2}}\mathbf{)}$	Neuron Body $\stackrel{\mathbf{~}}{\mathit{S}}\mathbf{=}{\mathit{a}}_{\mathbf{0}}\left({\mathit{x}}_{\mathbf{1}}\mathbf{+}{\mathit{x}}_{\mathbf{2}}\right)\mathbf{-}\frac{\mathbf{1}}{\mathbf{2}}{\mathit{a}}_{\mathbf{1}}{\left({\mathit{x}}_{\mathbf{1}}\mathbf{+}{\mathit{x}}_{\mathbf{2}}\right)}^{\mathbf{2}}$	Neuron Output $\mathit{O}\mathit{u}\mathit{t}=3\mathbf{tanh}(\tilde{\mathit{S}})$
(0,0)	0	0
(0,1)	0.523598775598299	1
(1,0)	0.523598775598299	1
(1,1)	0	0

), corresponding to the theoretically determined model $\left(a_0, a_1\right)=({\displaystyle \frac{\pi }{3}},-{\displaystyle \frac{\pi }{3}})$.

6. Conclusions

From the very inception of the classical neuron, the idea of forming a total summed signal in its body has been adopted:

$$S=\sum _{i=0}^n{w}_i{x}_i,$$

(112)

where $x_i$ and $w_i$ represent the inputs to the neuron and their respective weights $(i=0,1,\dots ,n)$. The neuron’s output then takes the value

$$Out=g\left(\sum _{i=0}^n{w}_i{x}_i\right),$$

(113)

where $g$ is its transfer function.

The linear combination of input signals and weights is the simplest and most intuitive way to model the interaction between inputs and the neuron. It is straightforward for mathematical representation and processing and allows for the use of established methods in linear algebra and optimization, facilitating the training of neurons and neural networks, especially in their early forms. The first stage where nonlinearity can emerge in neuron operation is with the application of the transfer function, which transforms the linear form (112) into the nonlinear signal (113). However, the linear sum alone, without an appropriate transfer function, cannot capture complex, nonlinear dependencies among the input signals. Additionally, it is sensitive to the scale of the input signals. If the inputs vary in scale, the sum can be dominated by larger values, potentially making the neuron less effective at processing information. This often necessitates pre-normalization of the inputs.

In this study, we set out to explore the possibility of changing the form of the signal (112). It turns out that if we normalize the input–output samples ${D=\left\{[x_{j0}=1,x_{j1},\dots x_{jn}; h_j]\right\}}_{j=1}^m$ within a sufficiently small interval, where $\mathsf{\lambda }=\max \left\{∆x_1, ∆x_2, \dots ,∆x_n\right\}\to 0$ for $∆x_i=x_i-x_{i-1}, i=1,2,\dots ,n$, we can consider a dynamic neural structure with a signal in its body as follows:

$$S=\sum _{i=0}^n{w}_i{x}_i={\int }_0^{{\displaystyle {\sum }_{i=0}^n}{x}_i}w\left(t\right)dt$$

(114)

where the function $w\left(t\right)$ can define each weight of the neuron as $w_i=w\left(x_i\right)$.

An even more interesting case emerged when the requirement for data proximity $\mathsf{\lambda }=\max \left\{∆x_1, ∆x_2, \dots ,∆x_n\right\}\to 0$ was not met. Under such conditions, we can consider new and unconventional artificial intelligence cell structures. Here, assuming that inputs are applied with constant weights $w_i=1$, an integral signal is formed:

$$\tilde{S}={\int }_0^{{\displaystyle {\sum }_{i=0}^n}{x}_i}w\left(t\right)dt,$$

(115)

where $w\left(t\right)$ is a continuous and smooth function (the neuron’s weight function). As a result, the axon of this type of neuron yields the following values:

$$Out=g\left({\int }_0^{{\displaystyle {\sum }_{i=0}^n}{x}_i}w\left(t\right)dt\right).$$

(116)

Thus, given the training set

$${D=\left\{[x_{j0},x_{j1},\dots x_{jn}; h_j]\right\}}_{j=1}^m, x_{j0}=1, \forall j=1,2,\dots ,m$$

(117)

the mean squared error is

$$E={\displaystyle \frac{1}{m}}\sum _{j=1}^m{\left(h_j-g\left({\int }_0^{{\displaystyle {\sum }_{i=0}^n}{x}_{ji}}w\left(t\right)dt\right)\right)}^2.$$

(118)

In the structure of integral neurons, traditional weights are absent, but in their place, we have the weight function $w\left(t\right)$, which not only replaces weights but also imparts a nonlinear character to the signal. With the introduction of the weight function, the optimization algorithms for the neuron are no longer aimed at finding a set of static weights but at discovering an appropriate form of $w\left(t\right)$ (its parameters) that minimizes the error. It turns out that gradient descent, Gauss–Newton, and the combined Levenberg–Marquardt optimization algorithms, when applied in this context, are convergent. During neuron training, starting from random parameters of the weight function, they are able to optimize it in a way that indeed leads to error minimization.

Are integral neurons (Figure 18) part of artificial intelligence? Absolutely. Like all artificial intelligence systems, integral neurons are initially unrefined and relatively ineffective in relation to the specific problem they are directed toward. However, during the training process, through the use of training samples, the parameters of their weight functions are adjusted in a way that drives the error toward its local or global minimum.

In the experimental part of this study, this type of integral neuron was used to solve the XOR logical function. It turns out that integral neural structures can be trained to solve this function—something that is impossible for standard neurons due to the linear nature of the sum in their bodies, which serves as the equation for the separating line. Theoretical solutions and experimental confirmations are shown for solving XOR using integral neurons with three different transfer functions. In each of the experiments, the weight functions are initialized with random parameters. Initially, the neurons have a high error rate, but after training with XOR samples, their weight functions are adjusted in a way that produces solutions with zero errors and parameters predicted by theory.

Integral neurons can also be used in more complex real-world situations, but the following limitations should be taken into account:

• Training Complexity: Training an integral neuron, especially with higher smoothness classes of the weight function, requires precise calculation of the gradient, Jacobian, and, potentially, Hessian matrix. This can potentially lead to significant computational challenges in real-world applications.
• Algorithm Convergence: As noted in the article, algorithms such as Gauss–Newton or Levenberg–Marquardt require appropriate initial parameters to ensure convergence to a global minimum. Otherwise, there is a risk of falling into a local minimum.
• Normalization Issues: To use the integral equivalent of a standard neuron, the following condition must be satisfied for the input data:

  $$∆x_i=x_i-x_{i-1}, i=1, 2,\dots ,n \mathrm{and} \lambda =\max \left\{∆x_1, ∆x_2, \dots ,∆x_n\right\}\to 0$$

This means that the input data must be carefully normalized to ensure the correctness of the integral representation and the execution of the algorithms. Of course, this is only necessary if we aim to replicate the behavior of standard neurons. In cases where we abandon requirement (1), we are dealing with an entirely new artificial neural structure for which such normalization is not necessary.

• Implementation: The practical implementation of the integral neuron requires complex computational systems for signal processing, especially when the integrand function has a non-trivial form.

A subject of future research is the comparison of the integral neuron with other types of neurons, e.g., rule-based neural networks where each neuron is a fuzzy inference system [68], as well as their capabilities to solve various regression and classification tasks.

Finally, it should be noted that in this study, integral neurons were equipped with polynomial weight functions. However, these could be functions of various types, as long as they meet the requirements for continuity and differentiability imposed by the convergence theorems of the training algorithms. Future goals in this area include exploring different types of weight functions and investigating the possibilities for building and using integral neural networks, as well as hybrid neural networks that incorporate both classical and integral neurons.