Neural Network Optimization Based on Complex Network Theory: A Survey

Abstract

Complex network science is an interdisciplinary field of study based on graph theory, statistical mechanics, and data science. With the powerful tools now available in complex network theory for the study of network topology, it is obvious that complex network topology models can be applied to enhance artificial neural network models. In this paper, we provide an overview of the most important works published within the past 10 years on the topic of complex network theory-based optimization methods. This review of the most up-to-date optimized neural network systems reveals that the fusion of complex and neural networks improves both accuracy and robustness. By setting out our review findings here, we seek to promote a better understanding of basic concepts and offer a deeper insight into the various research efforts that have led to the use of complex network theory in the optimized neural networks of today.

1. Introduction

Since the 1957 introduction by Rosenblatt [1] of the single-layer perceptron, the first artificial neural network model, which was inspired by the biological network of neurons in the human brain, the field of artificial neural networks has become one of the most important technologies in the era of Industry 4.0. The neural networks of today, compared with the simple perceptron, have become very “deep”, with more than 50 layers of network, compared to the simple one-layer network architecture of the perceptron. Deep neural networks (DNNs) [2] have become the key technology in all areas of research that use artificial intelligence. The success of DNNs as high-performance computational machines is due to their very large network topology that can process a large amount of data. For example, the convolutional neural network (CNN) [3], which is a popular DNN with a hierarchical structure similar to a digital image, has a total of 62,378,344 parameters (network weights) [4].

Complex network science is an interdisciplinary academic area based on mathematics, statistical mechanics, and computer science. Research topics in this field include complex network models, networks, and synchronization, as well as the use of complex networks in data science. It is logical that the powerful tools used in complex network theory might also aid research on how complex network topology affects performance in neural networks. Recent studies on the basic relationship between neural networks and complex network models are presented in [5,6,7,8]. However, a complete theoretical proof that explains topology-based neural network performance in terms of complex network theory has not yet been reported.

One of the first reviews of research works that investigated the influence of random topology in neural networks was published in [9]. Subsequently, the authors of [10] provided an overview of studies on the effects of complex network topology in neural networks, in terms of improved information storage and more efficient network connections. In contrast to these previous surveys, in this paper, we provide a detailed review of papers that describe the successful application of complex network models to different neural networks, to achieve improved performance in terms of accuracy and robustness. We considered the most important studies on complex network theory-based neural network optimization and application published between 2010 and 2023, and then carried out a comparative analysis based on input data optimization, complex network topology, neural network topology, and practical applications.

The remainder of the paper is organized as follows: Section 2 presents basic concepts related to complex and neural network models for complex systems theory-based neural network optimization research. In Section 3, we review various different methods proposed for improved performance of complex network-based neural networks. Section 4 offers a more detailed analysis of the methods described in Section 3, based on various specified criteria. Lastly, we offer some concluding remarks in Section 5.

2. Basic Concepts

2.1. Complex Network Model

In this section, we introduce important concepts related to complex network theory. First, we set out basic concepts related to graph representation that characterize the complex network model. We then cover the most important models, such as the random network model, the small-world network model, and the scale-free network model. The important complex network models are summarized in

Table 1. Description of complex network models.

Reference	Complex Network	Construction Method	Degree Distribution	Robustness Against Random Attack
[11,12]	Random network	Erdös and Rényi	Poisson	Strong
[13,14,15]	Small-world network	Watts and Strogatz	Poisson	Weak
[16,17,18]	Scale-free network	Barabási and Albert	Power law	Strong

on the basis of the construction method, degree distribution, and robustness. As shown in Table 1, both random networks and small-world networks have a Poisson degree distribution due to the homogeneous network topology.

2.1.1. Graph Representations

In order to represent the topological characteristics of a complex network model, it is described as a graph with ordered sets G = {V, E}, where V is a set of N vertices, and E is a set of edges connecting V nodes. For example, a graph G = {V, E}, with seven nodes and degrees K = 2, 3 (number of edges in a node), can be described with V = {1, 2, 3, 4, 5, 6, 7} and the set of edges E = { (1, 2), (1, 3), (2, 4), (3, 4), (4, 5), (5, 6), (6, 7)} [11,12]. Average path length is an important metric that represents the average distance between two vertices. This is determined by adding all the minimum distances between vertices i and j, which is then normalized by the number of all possible pairs of vertices. Another important metric is the clustering coefficient. This indicates the cliquishness of a typical friendship circle and is calculated by adding the actual number of links connecting the neighbors of node i. This is then normalized by the possible number of edges for i = 1 … N.

2.1.2. Random Network

A random network is a reference complex network model normally used to compare newly developed complex network models. A random network model is constructed with an initial ring of N nodes. Then, arbitrarily selected nodes i and j are connected with connection probability p, and this step is repeated for all possible node pairs N(N − 1)/2. The degree distribution of a random network is characterized by a Poisson distribution with parameters of degree k and average degree <k>.

2.1.3. Small-World Network

A small-world network is a complex network with a small average path length and robust topology [13,14,15]. As with the random network, the model starts with a ring of N nodes. However, to construct a small-world network, all the connected links are rewired, with rewiring probability, p, for the randomly chosen nodes i and j. The rewiring process leads to a decrease in average path length, as a few long-range links create shortcuts.

2.1.4. Scale-Free Network

The main characteristic of a scale-free network is that it is an evolutionary network that grows with the dynamic addition of new nodes [16,17,18]. The network construction procedure begins with a small number of m₀ nodes with degree K. A new node is introduced into the network and connected to m existing nodes using a preferential attachment process based on maximum degree probability. The degree distribution of a scale-free network exhibits the power law property with a power exponent of α > 1. This property is due to the non-negligible likelihood of finding nodes with a very high number of connections (hubs). Real network with power-law distributions are email networks, the Internet, and protein interactions.

2.2. Neural Network Model

In this section, we introduce the important neural network models used in the recent studies of complex network-based neural networks reviewed in Section 3 and Section 4. To this end, we now describe the basic concepts of the multilayer perceptron (MLP), restricted Boltzmann machine (RBM), convolutional neural network (CNN), graph neural network (GNN), and spike neural network (SNN).

2.2.1. Multilayer Perceptron

The multilayer perceptron (MLP) has an architecture that consists of three layers called the input layer, hidden layer, and output layer, as shown in Figure 1. The input layer receives input data that are used as a basis for prediction, classification, and regression. The hidden layer maps the input layer to a higher dimension using various nonlinear activation functions. The weights between the nodes of each MLP layer are trained to minimize error using a backpropagation algorithm.

2.2.2. Restricted Boltzmann Machine

Restricted Boltzmann machines (RBMs) are stochastic neural networks that consist of a visible layer and a hidden layer. The joint probability distribution of these two layers is learnt, as shown in Figure 1. RBM weights are trained using a contrastive divergence algorithm that approximates the gradients of different functions [19,20]. RBMs can be stacked and used as an unsupervised pretraining module in a deep neural network (DNN) called a deep belief network (DBN) that initializes the weights of the MLP module within the DBN.

2.2.3. Convolutional Neural Network

The convolutional neural network is one of the most famous DNNs. It involves an input layer, convolutional layer, pooling layer, fully connected layer, and output layer, as shown in Figure 1. The convolutional layer extracts different features from image data using different kinds of kernels [21]. The pooling layer downsamples the convolutional layer output using maximum values or average values. Lastly, the fully connected layer and the output layer—with classical MLP-like structure—make the final decision based on the data processed by multiple convolutional and pooling layers.

2.2.4. Graph Neural Network

Graph neural networks (GNNs) are neural networks that are applied to graph-based data. The basic modules are the propagation module, sampling module, and pooling module [22]. The propagation layer aggregates information between nodes to capture topological features of the graph data, as shown in Figure 1. The sampling layer is used reduce the amount of node and layer information. Lastly, as with the CNN, the pooling layer is used to reduce graph representations. GNN applications include natural language processing, computer vision, and chemical graph structure analysis.

2.2.5. Spiking Neural Network

Spiking neural networks (SNNs) are neural network models that have been developed to model the behavior of the biological neural network more realistically than conventional artificial neural network models [23]. By applying the integrate-and-fire neuron model, SNN emulates the asynchronous action potential, or spike, of biological neural networks, as shown in Figure 1. The output spike in SNN is computed by integrating the input spike to neuron j at layer l, and it is activated when it crosses the firing threshold.

3. Studies of Improved Performance Achieved by Complex Network-Based ANNs

Zheng et al. [24] proposed the application of small-world topology to fully connected neural networks, using the matrix decomposition and synaptic connection elimination methods. The key to the proposed method was to define a condition in finding unimportant synaptic connections and then control the number of link eliminations. The performance evaluation was carried out by comparing the Chinese character recognition accuracy achieved by fully connected neural networks with that achieved by small-world neural networks, with 65.7% and 78.7% connection eliminations, respectively. The small-world characteristics of the constructed neural networks were verified by observing the Poisson distribution of the degree distribution of the reconstructed network link structure. According to this experiment, the authors showed that the small-world neural network was able to perform as effectively as fully connected neural networks, with a lower number of links.

Li et al. [25] introduced a neural network controller system-based small-world network model for application in an electrohydraulic actuation system. In contrast to the authors of the study just described, who used a link-elimination-based small-world neural network construction, Li et al. rewired the regular neural network’s links with the rewiring probability p. For the application of the small-world neural network as an adaptive controller, the number of hidden layers was set to two, and the numbers of neural units in the first and second layers were both set to six. The neural network structure was, thus, set to 3–6–6–1, representing the number of neural units from the input layer to the hidden layer to the output layer. As for the rewiring probability, to construct the small-world topology in the proposed model, a value of p = 0.1 resulted in 30% better performance than the regular neural network in a random disturbance environment. In summary, the small-world topology-based neural network achieved superior performance, compared with regular topology-based neural networks, when acting as an adaptive controller for a direct-drive electrohydraulic actuation position control system.

Mocanu et al. [26] presented a novel evolutionary algorithm called the sparse evolutionary training (SET) procedure to construct a neural network with sparse topology. In the proposed SET algorithm, weights close to zero were removed, and new weights were randomly added to maintain a fixed number of layers and neural units. To evaluate the effectiveness of the proposed algorithm, it was applied to a restricted Boltzmann machine (RBM), a multilayer perceptron (MLP), and a convolutional neural network (CNN). The structure of all the neural networks evolved to produce a scale-free topology with a power-law distribution. In the experiment carried out to test SET-based CNN performance, the CNN architecture was chosen to have a conv(32, (3,3))–conv(32, (3,3))–pooling–conv(64, (3,3))–conv(64, (3,3))–pooling–conv(128, (3,3))–conv(128, (3,3))–pooling architecture, where the brackets represent (number of filters, (kernel size)). The CIFAR10 dataset was used to compare the accuracy performance of the fully connected CNN and the SET-CNN. It was shown that CNN achieved 87.48% accuracy, but SET-CNN achieved 90.02% accuracy, with only 4% of the original number of neural connections.

Ribas et al. [27] investigated a novel approach for applying complex network theory to a randomized neural network (RNN) for the purpose of texture analysis. To enhance the performance of the RNN, the input texture image was modeled as a complex network with pixels and neighbor topologies represented by vertices and edges. The RNN was trained on the basis of the degree of vertex from the complex network transformed input image, and the grayscale of the pixel. The weights of the RNN output layer were used as the texture signatures. The authors used the grayscale texture databases Brodatz, Outex, USPTex, and Vistex to evaluate the proposed complex network-based RNN. The experimental results showed that the proposed method achieved accuracy values of 91.54%, 96.95%, 96.11%, and 99.19 %, respectively, for the four test texture databases, representing a level of performance exceeding that of the 17 conventional texture analysis systems tested.

Wang et al. [28] proposed two small-world neural networks based on Watts–Strogatz and Newmann–Watts small-world construction methods. These were termed the Watts–Strogatz backpropagation (WSBP) neural network and the Newmann–Watts backpropagation (NWBP) neural network. The initialized WS and NW neural networks were trained using the classical feed-forward propagation and feedback propagation algorithms to minimize errors. In an experiment carried out to forecast wind power, real-world 2017 data from a wind farm was used. The number of hidden layers was set to five, with five neurons in each layer for both WSBP and NWBP neural networks. The initial weights were set as [−0.3, 0.3] with p = 0.2. In terms of the experimental goal of predicting wind power over a 24 h timescale, SVM and NWBP achieved 12.732% and 12.482% prediction errors, respectively, which represented a 9% improvement in error performance, compared with SVM and BP methods.

Huang et al. [29] introduced enhanced convolutional neural networks (CNNs) based on the Erdos–Renyi (ER) random network model, Watts–Strogatz (WS) small-world network model, and the Barabasi–Albert (BA) scale-free network model, to carry out medical imaging classification of brain tumors. CNN models based on complex networks (CNNBCN) are generated using undirected random graphs such as random, small-world, and scale-free networks. The random graph is first converted to a directed acyclic graph (DAG); afterward, a data stream module is added with operation modules such as deconvolution, batch normalization, and activation functions. With these steps, a final module is created that can be used in multiple groups, as in the proposed CNNBCN architecture. Figure 2 shows an example of CNNBCN structure with four modules, each consisting of input, hidden, and output layers. The red circles in the figure represent the input and output nodes connecting the multiple random graph networks. The final output of the last or fourth random graph network is passed to the classifier. Huang et al. evaluated their proposed CNNBCN on the basis a medical dataset obtained from 233 patients and consisting of 708 meningioma images, 1426 glioma images, and 930 pituitary tumor images. In the experiment, the CNNBCN consisted of five modules, with 32 nodes in each module. The proposed CNNBCN achieved very high accuracy levels of 100% for the training dataset and 95% for the test dataset, and with a reduced training time, compared with ResNet-151 and DenseNet-161.

Gao et al. [30] investigated the use of complex network theory in a graph convolutional neural network (GCN). They sought to extract topological features of graph-based data which were more relevant to data labels and, thus, improve classification performance. Their proposed topological GCN (T-GCN) consisted of three convolutional modules based on a topology graph, an original graph, and node features that were extracted from the original graph features. Finally, an attention mechanism was applied to learn the embedding weights of the features obtained from the previous convolution modules. To evaluate the proposed T-GCN, the authors used the following datasets: Citeseer, UAI 2010, ACM, BlogCatalog, and Flickr. In addition, to compare classification performance, the following methods were used: Deep Walk, LINE, Chebyshev, GCN, KNN–GCN, Cluster-GCN, GAT, DEMO-Net, and AM-GCN. The experimental results showed that, on the BlogCatalog and Flickr datasets, the proposed T-GCN achieved accuracy and F1 scores which improved upon those achieved by conventional methods by 8.71% and 1.84%, respectively. Similar improvements were also recorded for other datasets.

Lee et al. [31] proposed the application of a multilayer perceptron neural network (MLPNN) to reconstruct damaged complex networks under random attacks. The proposed reconstruction method was based on a network link information matrix containing minimal network structure information. The MLPNN used in the proposed system was trained using the link information matrix of the damaged networks and corresponding desired original network link information matrix. The goal of the proposed method was to identify the original network, given the link list data of the damaged network. The MLPNN structure was set to 45/435/1225–64–4–8, representing the number of units from the input layer, to two hidden layers, to output layer. The input dimensions were set to 45, 435, and 1225, representing all possible node pair combinations. The output dimension was set to eight, to represent the maximum number of 256 possible original networks. The experiment was carried out using a metric called probability of reconstruction error, which measured the total number of existing links in the damaged networks, compared with the link difference number in the reconstructed network. It was shown that the proposed MLPNN-based reconstruction system could reconstruct around 70% of the original network topology for 10% of node failure cases in small-world networks, and around 50% in scale-free networks.

Wei et al. [32] introduced a graph convolutional network (GCN) to achieve robust defense against topology attacks. The first step in their proposed method was to divide the attacked graph data into attribute (node) and topology (edge) subgraphs. The attribute subgraph was used as initial graph data to train the classifier based on double-layer GCN. The topology subgraph was used as the topology mask for the topology generator based on triple-layer GCN. The proposed method reconstructed the damaged graph data through outer topology reconstruction iteration and inner classification iteration. To evaluate the proposed method, three citation datasets—Cora_ml, Citeseer, and Pubmed—were used with two attack models: meta-attack and random attack. In the experiment, the classifier was implemented with 16 hidden units and a learning rate equal to 0.01. The proposed method was able to reconstruct 80% of damaged graph data for all the attack models considered.

Guo et al. [33] presented robust spiking neural network (SNN) models based on small-world network topology and scale-free network topology. These were termed small-world SNN (SWSNN) and scale-free SNN (SFSNN). The goal of this study was to evaluate the anti-injury abilities of SWSNN and SFSNN against random attacks. To construct the SWSNN and SFSNN, an Izhikevich neuron model was used as the node of the complex network-based SNN, and a synaptic plasticity model with excitatory and inhibitory synapses was used for the edges. The SWSNN was constructed to have small-world network topology with rewiring probability p = 0.2, and the SFSNN was constructed to have scale-free topology with a power-law distribution, with the power-law index η equal to 2.154. To evaluate the robustness of the proposed SWSNN and SFSNN models, the relative change in the firing rates before and after the attacks was used. The firing rate in SNN was defined as the number of spikes emitted by the neurons per unit time in the SNN. The experimental results showed that SFSNN had the stronger anti-injury ability of the two models, with a change rate of 0.64–4.11% compared to SWSNN’s rate of 0.40–8.94%. Nevertheless, both SFSNN and SWSNN exhibited strong anti-injury ability under random attacks with information processing functions.

Guo et al. [34] investigated the robustness of SWSNN and SFSNN under targeted attack by extending the work carried out in [33]. A synaptic plasticity model was used to create edges that play an important role in nerve system regulation. The targeted attack model was designed to be grouped into high-, intermediate-, and low-degree node attacks. Again, the relative change in firing rates was used as the metric to evaluate the robustness of SWSNN and SFSNN against targeted attacks. The experimental results revealed small differences in robustness in intermediate- or low-degree node attack scenarios. In short, with the exception of high-degree node attack cases, SWSNN exhibited better anti-injury performance, compared with SFSNN.

Kaviani et al. [35] proposed a novel defense method for neural networks against backdoor attacks based on scale-free networks called link pruning with scale freeness (LPSF). This method consists of three main steps: data enumeration, link pruning, and scale-free reconstruction. In the first step, dormant or less active pixels are identified that can act as a site where a trigger can be attached. In the second step, links (neurons) related to these redundant pixels that can contain triggers are removed. In the final step, the neural network topology is reconstructed into a scale-free structure to strengthen the accuracy performance that might have degraded due to the link pruning process. Kaviani et al. verified the robustness of this proposed defense method using MNIST, FMNIST, and HODA datasets that have backdoor attack triggers with one-pixel, four-pixel, nine-pixel, 12-pixel patterns. Accuracy was measured in terms of the correct classification of clean data. Attack success rate was measured in terms of target label classification over the malicious dataset. It was found that LPSF achieved an improved attack success rate, compared with the fine pruning method (one of the first defense methods against backdoor attack), while maintaining high accuracy performance. One important point to notice is that established methods are designed as defense mechanisms to be applied after the attack has happened, while LPSF is designed as a pre-attack solution, based on input data patterns and neural network structure optimization, using scale freeness.

4. Discussion

In this section, we study the general framework of the complex network-based neural network models used in the 12 important research works from 2010 to 2023 described in the previous sections of this paper. To aid understanding of this general framework,

Table 2. Descriptions of complex network-based neural networks.

Ref.	Complex Network	Neural Network	Input Data	Key Technology	Applications
[24]	SWN	MLP	Chinese character	Synaptic connection elimination	Chinese character recognition
[25]	SWN	MLP	Position control signal	ANN link rewiring	Electrohydraulic control system
[26]	SFN	MLP, RBM, CNN	MNIST, FMNIST, CIFAR10, HIGGS	Sparse evolutionary training	Image recognition
[27]	RN	RNN	Brodatz, Outex, USPTex, Vistex	Image to CN topology mapping	Texture image recognition
[28]	SWN	MLP	Wind-related data: speed, direction, power	WS- and NS-based CNNN construction	Wind power forecasting
[29]	SWN, SFN	CNN	Medical dataset: meningioma, glioma, pituitary tumor	ER-, WS-, BA-based CNNN construction	Brain tumor image classification
[30]	RN	GCN	Citeseer, UAI2010, ACM, BlogCatalog, Flickr	Graph feature-based CNN	Topological features extraction
[31]	SWN, SFN	MLP	Complex network topological features	Network topology extraction	Damaged network reconstruction
[32]	RN	GCN	Cora-ml, Citeseer, PubMed	Attribute subgraph generation	Damaged network reconstruction
[33]	SWN, SFN	SNN	Neural network attack	Graph feature-based SNN	Anti-injury performance
[34]	SWN, SFN	SNN	Neural network attack	Graph feature-based SNN	Anti-injury performance
[35]	SFN	MLP	MNIST, FMNIST, HODA, Oracle-MNIST	Link pruning-based SFN construction	ANN defense against attack

sets out these models that improved the robustness and accuracy of different neural network architectures, according to various complex network techniques. These are discussed in Section 4.1, Section 4.2 and Section 4.3 in terms of their input data optimization, complex network topology, and neural network applications, respectively. It is recommended that the readers use the information given in Table 2 to compare the 12 research works that had the same of goal of optimizing the network and data topology.

4.1. Input Data Optimization

One of the many important methods used to improve neural network performance is input data optimization [36,37,38]. This is due to the close relationship between the training data and the trained neural network structure that includes weight distribution, node activation pattern, and overall connection topology. Data such as images and text can be mapped into graph data where each pixel and word is represented as a node. These graph-transformed data can be applied to various graph neural networks with improved performance. Many of the studies reported in Section 3 applied complex network optimization tools to map input data into graph topology. In [28], the authors mapped input texture images into a complex network with nodes and links to be trained in recurrent neural networks. Each pixel in the texture image is mapped to vertex information, and the difference of two neighbor pixels is mapped to directed edge information. In [31], the authors designed three graph convolutional networks (GCNs) to process a topology graph, an original graph, and node features to be used in an attention-based system. These three GCNs are independently designed modules to maximize extraction of three different feature information to be fused by the attention mechanism toward final classification. Further research on complex network theory-based input data optimization is needed to obtain a standard method in data preprocessing for neural network performance enhancement.

4.2. Complex Network Topology

Among the studies reviewed here, the complex network topologies used to optimize neural networks and data topology were small-world network (SWN), scale-free network (SFN), or general random network (RN) topologies. The majority of the works constructed complex network topologies in neural networks by means of a two-step method. The first step is to initialize the neural network using conventional fully connected topology methods. This is followed by the restructuring of links into SWN, SFN, or RN topologies. We found that SWN was the most popular method for performance improvement of neural network models, while SFN offered more enhanced robustness against external attacks. However, there are as yet no optimum decision guidelines for choosing the right complex network topology for targeted improvements in either performance or robustness.

4.3. Complex Network Neural Network Applications

Considering the works surveyed in the previous sections, it seems that there were no limitations on applying complex network topologies [39,40,41] in various neural network models to enhance performance or robustness. Neural networks that were optimized in the 12 important research works were as follows: multilayer perceptron, restricted Boltzmann machine, convolutional neural network, graph neural network, spiking neural network, and recurrent neural network. We can, therefore, state that any neural network consisting of layers of nodes connected through complex links can be optimized using various complex network topologies. For example, in [30], CNN topology was enhanced using RN, SWN, and SFN for the medical imaging classification of brain tumors, resulting in high accuracy performance. However, researchers have not yet provided a theoretical relational mapping of complex network topology to neural network topology.

5. Conclusions

In this paper, we provided basic concepts of complex network models, including graph representations, random networks, small-world networks, and scale-free networks. We also presented a brief overview of important neural networks to promote understanding of contemporary complex system theory-based optimization works in neural network modes. To compare the different aspects of these 12 important works, we distinguished between those studies which sought performance improvement and those which sought to defend neural network topology against attack. To gain insight into the different optimization procedures developed in these works, we evaluated them according to complex network topology, neural network topology, input data used, key technology, and applications. The main findings of this paper are summarized as follows:

• Complex network optimization tools can be applied to big data to be used for neural network training and directly to neural network structure with improved performance in accuracy and robustness.
• However, simultaneous joint optimization approach targeting both the data and the neural network using complex network theory was nonexistent. It is of our opinion that further improvement in performance, compared to the conventional methods, is possible by considering both the input/output data structure and the target neural network structure.
• To stimulate future interest in the field of complex network theory-based neural networks research, more work should be carried out in other application areas such as AI-based self-driving.