A Hybrid Sparrow Search Algorithm of the Hyperparameter Optimization in Deep Learning

Abstract

Deep learning has been widely used in different fields such as computer vision and speech processing. The performance of deep learning algorithms is greatly affected by their hyperparameters. For complex machine learning models such as deep neural networks, it is difficult to determine their hyperparameters. In addition, existing hyperparameter optimization algorithms easily converge to a local optimal solution. This paper proposes a method for hyperparameter optimization that combines the Sparrow Search Algorithm and Particle Swarm Optimization, called the Hybrid Sparrow Search Algorithm. This method takes advantages of avoiding the local optimal solution in the Sparrow Search Algorithm and the search efficiency of Particle Swarm Optimization to achieve global optimization. Experiments verified the proposed algorithm in simple and complex networks. The results show that the Hybrid Sparrow Search Algorithm has the strong global search capability to avoid local optimal solutions and satisfactory search efficiency in both low and high-dimensional spaces. The proposed method provides a new solution for hyperparameter optimization problems in deep learning models.

1. Introduction

The abilities of data collection, storage, and processing in scientific research and engineering applications have been greatly improved with the rapid development of technologies [1]. Deep learning has been introduced in different fields of science and engineering, especially in data processing and analysis [2]. Based on artificial neural networks, deep learning imitates the mechanism of the human brain to process data information such as images, sounds, and texts. Deep learning has shown its potential in having great learning ability, wide coverage, high adaptability, and excellent portability in data processing and analysis [3].

Deep learning uses a network structure with multiple layers. Each layer processes the received signal and passes it to the next layer. In a deep neural network, many layers can be used between the input and output layers. These layers can perform linear and non-linear data transformations [4].

Hyperparameters are parameters that need to be determined before forming a neural network, such as the batch size, learning rate, and dropout rate. The performance of a deep learning model will be greatly impacted by the hyperparameter configuration.

In general, building an effective deep learning model is a complex and time-consuming process that involves determining an appropriate structure of a deep learning model and its hyperparameters. The performance of a deep learning algorithm is greatly affected by its hyperparameters. Therefore, determining hyperparameters is an important task in applications of deep learning. The purpose of this paper is to propose a hyperparameter optimization algorithm with superior performance.

Different methods have been applied in setting hyperparameters for deep learning, such as the manual search, grid search, and random search. However, these methods have the problems of poor performance in high-dimensional models, inefficiency, and low accuracy. Therefore, an effective method is required for the optimization of hyperparameters for deep learning algorithms.

Hyperparameter optimization finds a set of hyperparameters to fit the needs of a deep learning model. The search is conducted using a mathematical model of optimization [5]. The hyperparameter optimization is normally considered a “black box” searching process to determine the hyperparameter configuration of a deep learning model [6].

This paper proposes a method for hyperparameter optimization, by combining the Sparrow Search Algorithm and Particle Swarm Optimization, called the Hybrid Sparrow Search Algorithm. This method uses the advantages of the avoidance of the local optimal solution in the Sparrow Search Algorithm, and the search efficiency of Particle Swarm Optimization, to achieve global optimization.

The contributions of this research are as follows: (1) development of a heuristic algorithm with a strong global search ability; (2) improvement of the hyperparameter optimization; (3) application of the proposed algorithm in neural networks.

The remainder of this paper is organized as follows. Section 2 introduces the existing research. Section 3 provides details of the proposed method. Experiments to verify the proposed method are discussed in Section 4. Section 5 presents a discussion and future work, and the conclusion of this paper is presented in Section 6.

2. Related Research

2.1. Hyperparameter Optimization

Different methods have been proposed for the hyperparameter optimization including manual search, grid search, random search, and Bayesian optimization.

Manual search [7] can determine hyperparameters for simple models. Before the emergence of the big data era, neural network models used to process data were generally not complicated. Hyperparameters of a model could be manually decided by field experts. However, with big data applications, there are increased hyperparameters in a neural network model, for which manual search is not able to meet the demand [8]. Automatic hyperparameter design has become a new research field.

Larochelle [9] proposed a grid search algorithm to balance the system performance and computation efficiency in determining hyperparameters. Grid search is an exhaustive method that takes a compromise between computational overhead and performance. For a large dataset, it has an “exponential explosion” problem [10] with reduced search efficiency. Bergstra [11] proposed a random search method that is simple and easy to use. However, this method is largely blind and has low adaptability. It is not a high-performance hyperparameter search method.

For sequential models [12,13], Bayesian optimization [14,15,16,17] is one of the most classic methods of hyperparameter optimizations. Compared with manual and random search methods, Bayesian optimization can fully use the information of previous searches in solving some complex problems [18]. However, this method can easily fall into a local optimal solution as it samples only around the optimal point.

Heuristic algorithms are based on experience that provides a feasible solution to the problem at an acceptable cost (computing time and space) [19]. They have been widely used in hyperparameter optimization. Sun [20] used a Simulated Annealing (SA) algorithm with a fast convergence rate for clustering problems of neural networks. Zhang [21] and Francescomarino [22] used Genetic Algorithms (GAs) to optimize network models based on ideas of heredity, crossover, and mutation in practical problems. As an improved evolutionary algorithm, Covariance Matrix Adaptation Evolutionary Strategies (CMA-ESs) [23] have been used to optimize neural networks. Lorenzo [24] and Djenouri [25] used Particle Swarm Optimization (PSO) with a fast convergence rate to optimize a deep learning network. However, these heuristic algorithms have a common problem, namely, that they easily fall into a local optimal solution when the objective function is non-convex. They are not effective in applications.

There are some recently proposed hyperparameter optimization methods. Ozturk [26] used a stochastic gradient descent to optimize the echo state network. Hu [27] used the Chimp Optimization Algorithm to increase the reliability and real-time capability of the network for classifying chest X-ray images. Kalita [28] developed Moth flame optimization and knowledge-based-search to optimize the hyperparameters of Support Vector Machine (SVM) [29]. Wu [30] used the sine–cosine algorithm to tune parameters of the Extreme Learning Machine (ELM) for diagnosing COVID-19 positive cases. Wang [31] used Whale Optimization for a real-time COVID-19 detector with parallel implementation capability. Several other algorithms have also been used to diagnose COVID-19 in recent years [32,33,34]. However, these methods only process some specific models.

In summary, the existing hyperparameter optimization algorithms generally have the problems of poor global search ability and propensity to fall into local optimal solutions [35,36].

2.2. Particle Swarm Optimization (PSO) and Sparrow Search Algorithm (SSA)

PSO [37] is an evolutionary algorithm inspired by the regularity of bird swarm activities. Based on the bird swarm activity behavior, PSO shares information about individuals in the entire swarm, for the evolution process from disorder to order in a problem space, to obtain the optimal solution.

PSO is an optimization algorithm based on iteration. The system is initialized as a set of random solutions to iteratively search for the optimal value. The algorithm searches through particles following the optimal particle in the solution space. Although this algorithm has a high search efficiency, it cannot avoid local optimal solutions when all individuals are concentrated near a local optimal solution.

The Sparrow Search Algorithm (SSA) [38] uses heuristic search to simulate the foraging process of sparrows as a kind of discoverer–follower model with a scouting and early warning mechanism. SSA has the ability to avoid the local optimal solution. However, it converges slowly and cannot obtain the optimal solution within an acceptable time [38]. In addition, our research found that this algorithm has data overflow problems caused by excessive exponents when the solution value is large. As a result, a large number of solutions are concentrated on the upper bound of the feasible region, which reduces the diversity of solutions and affects the algorithm performance. In addition, SSA has not been used to optimize hyperparameters of the neural network model.

In summary, the existing methods generally have poor global search ability. Although SSA has the ability to avoid local optimal solutions, it cannot be used in hyperparameter optimization problems due to the problem mentioned above. As a swarm intelligence algorithm, PSO is efficient for complex problems such as hyperparameter optimization. However, it also easily falls into a local optimal solution. This paper proposes a Hybrid Sparrow Search Algorithm (HSSA) having the advantages of both SSA and PSO for strong global search ability in complex problems to obtain the optimal solution within an acceptable time for hyperparameter optimization.

3. Proposed Approach

3.1. Hybrid Sparrow Search Algorithm (HSSA)

In order to address the problem of the existing hyperparameter optimization methods, which easily fall into local optimal solutions, we combine SSA with PSO, termed HSSA, by taking advantage of the velocity and displacement formula in PSO on the basis of SSA. The details are as follows.

(1)

Algorithmic Modeling

Among sparrows, individuals with a high fitness value act as discoverers, and other individuals act as followers. At the same time, a certain proportion of individuals in the population is selected for detection and early warning. If any danger is found, they will search alternatives. The population with n sparrows is as follows:

$$X=\left. [\begin{array}{cccc}{x}_1^1& x_1^2& \dots & x_1^m\\ x_2^1& x_2^2& \dots & x_2^m\\ \dots & \dots & \dots & \dots \\ x_n^1& x_n^2& \dots & x_n^m\end{array}\right]$$

(1)

where m represents the dimension of variables to be optimized, and n is the number of individuals. m depends on the dimension of the problem. n represents the size of the population. In general, a large n leads to high population diversity and high optimization accuracy, but the iteration speed is slow. The fitness value of all sparrows can be expressed as follows:

$$F=\left. [\begin{array}{c}f(\begin{array}{cccc}{x}_1^1& x_1^2& \dots & x_1^m\end{array})\\ f(\begin{array}{cccc}{x}_2^1& x_2^2& \dots & x_2^m\end{array})\\ \dots \\ f(\begin{array}{cccc}{x}_n^1& x_n^2& \dots & x_n^m\end{array})\end{array}\right]$$

(2)

where f is the fitness value. F contains the fitness of all individuals in the entire population.

(2)

Basic Rules

Discoverers usually have high energy reserves and are responsible for searching for food-rich areas. They provide foraging areas and directions for all followers. The level of energy reserve depends on the fitness value of every individual.

As soon as predators are detected, sparrows begin to chirp to send alarm signals. If the alarm value is greater than the safe value, discoverers will take followers to other safe areas.

Discoverers and followers change dynamically. As long as a better source of food can be found, a sparrow can become a discoverer, but the proportion of discoverers and followers in the entire population remains unchanged. In other words, whenever a sparrow becomes a discoverer, another sparrow becomes a follower.

Followers with less food have poor foraging positions in the entire population. Hungry followers are more likely to fly to other places to get food.

During the foraging process, followers can always search for the discoverer who provides the best food, or forage around the discoverer. At the same time, in order to increase their food reserves, some followers may constantly monitor discoverers for food resources.

Once aware of the danger, individuals at the edge of population will quickly move to a safe area for better positions. Individuals located in the middle of the population will randomly fly to other sparrows.

(3)

Discoverers

Discoverers account for 10–20% of the entire population. Location updates of the discoverers are given as follows:

$$X_{i,j}^{s+1}=\left\{\begin{array}{l}{X}_{i,j}^s\cdot \exp (-\frac{i}{\alpha \cdot s_{\max }}),\begin{array}{ll}& R<T\end{array}\\ X_{i,j}^s+g\cdot L,\begin{array}{ll}\begin{array}{ccc}& & \begin{array}{cc}& \end{array}\end{array}& R\ge T\end{array}\end{array}\right.$$

(3)

where i, j, and s are the ith sparrow, jth dimension, and sth iteration, respectively. X represents location information. smax is the maximum number of iterations. α (α ∈ (0, 1]) is a random number. R (R ∈ [0, 1]) and T (T ∈ [0.5, 1]) represent the warning value and safety value, respectively. g is a random number of a normal distribution. L represents a 1 × m matrix, and each element in the matrix is 1. Generally, the larger the value of s, the better the optimization effect; however, it takes more time. The likelihood of an individual being frightened depends on T.

(4)

Followers

All sparrows except discoverers are followers. The original location update formula in SSA is as follows:

$$X_{i,j}^{s+1}=\left\{\begin{array}{l}g\cdot \exp (\frac{X_w^s-X_{i,j}^s}{i^2}),\begin{array}{ccc}& & \end{array}\\ X_P^{s+1}+\left|X_{i,j}-X_P^{s+1}\right|\cdot A^{+}\cdot L,\end{array}\begin{array}{cc}& i>n/2\\ & i\le n/2\end{array}\right.$$

(4)

where X_w is the worst position. Our research found that Equation (4) has data overflow problems caused by excessive exponents in the case i > n/2 when the solution value is large. For example, if X_sw = 5000, X^s_i,j = 1000, i = 4. X^s_i,j = g × exp(250). Obviously, this causes a data overflow problem.

In order to solve this problem and improve the global search ability, we combine Equation (4) with the velocity and displacement formula in PSO. The new location updates of followers are as follows:

$$V_{i,j}^{s+1}=\omega \cdot V_{i,j}^s+c_1\cdot r_1\cdot (B_{i,j}^s-X_{i,j}^s)+c_2\cdot r_2\cdot (B_{g,j}^s-X_{i,j}^s)$$

(5)

$$X_{i,j}^{s+1}=\left\{\begin{array}{c}{X}_{i,j}^s+V_{i,j}^{s+1},\begin{array}{cccc}& & & \begin{array}{cc}& \end{array}\end{array}\\ X_P^{s+1}+\left|X_{i,j}-X_P^{s+1}\right|\cdot A^{+}\cdot L,\end{array}\begin{array}{cc}& i>n/2\\ & i\le n/2\end{array}\right.$$

(6)

where V represents speed. ω (ω ∈ [0, 1]) is inertia weight. c₁ and c₂ are learning factors that generally take values of 0–4. r₁ and r₂ are random numbers between 0 and 1. B_i is the historical optimal solution of the ith sparrow. B_g is the global optimal solution of the entire population. X_p is the best position occupied by the discoverers. A represents a 1 × m matrix, and each element in the matrix is 1 or −1, and A⁺ = A^T(AA^T)⁻¹. ω is non-negative. In general, when it is large, the global search ability is strong; when it is small, the local search ability is strong. c₁ and c₂ are the individual learning factor and social learning factor of each individual, respectively. Nandar’s [39] experiment showed that satisfactory solutions can be obtained when c₁ and c₂ are constants; usually, c₁ = c₂ = 2.

Equation (5) is the speed update formula. Equation (6) is formed by adding the speed update formula to the position update formula of followers. This improvement not only solves the data overflow problem, but also improves the search speed. It enables HSSA to search for the hyperparameter optimization in an acceptable time.

(5)

Vigilantes

Location updates of vigilantes are as follows:

$$X_{i,j}^{s+1}=\left\{\begin{array}{l}{B}_{g,j}^s+\beta \cdot \left|X_{i,j}^s-B_{g,j}^s\right|,\begin{array}{ccc}& & f_i>f_g\end{array}\\ X_{i,j}^s+k\cdot (\frac{\left|X_{i,j}^s-X_w^s\right|}{(f_i-f_w)+\epsilon }),\begin{array}{cccc}& & & f_i\le f_g\end{array}\end{array}\right.$$

(7)

where X_w is the worst position. β is a parameter for controlling the step length. It is a random number that obeys the standard normal distribution. k is a random number between −1 and 1. f_i is the fitness value of the current sparrow. f_g and f_w are global best and global worst fitness values, respectively. ε is a constant to avoid zero in the denominator.

(6)

Algorithm Framework

HSSA is composed of SSA and PSO. As shown in Figure 1, the left side of the flowchart is the SSA part, and the right side is the PSO part. The SSA part includes the position calculation of discoverers, followers, and vigilantes. The PSO part calculates the position of followers when i > n/2. The steps of HSSA are as follows.

Firstly, the initial population is randomly generated. Secondly, a fitness function is determined to evaluate the fitness of each individual and update the best solution. Thirdly, the population is divided into discoverers and followers based on fitness to update positions of discoverers and followers. If i > n/2, the PSO part is used to calculate the position of followers. Vigilantes are then randomly generated to update their position. Finally, the global best solution is calculated to determine whether the ending condition is met. The above steps are repeated until an individual that meets the ending condition is found. An individual who meets the ending condition is considered the best solution of HSSA. The pseudo-code is shown in Algorithm 1.

Algorithm 1. Procedure Hybrid Sparrow Search Algorithm
1	Input: individuals n, dimension m, iterations smax
2	Output: optimal value
3	Initialize the population n, individual optimal value fi and global optimal value fg
4	for s in smax do
5	Divide the population n into discoverers nd and followers nf
6	for i in nd do
7	for j in m do
8	if R < T then
9	$X_{i,j}^{s+1}=X_{i,j}^s\cdot \exp (-\frac{i}{\alpha \cdot s_{\max }})$
10	else
11	$X_{i,j}^{s+1}=X_{i,j}^s+g\cdot L$
12	for i in nf do
13	for j in m do
14	if i > n/2 then
15	$V_{i,j}^{s+1}=\omega \cdot V_{i,j}^s+c_1\cdot r_1\cdot (B_{i,j}^s-X_{i,j}^s)+c_2\cdot r_2\cdot (B_{g,j}^s-X_{i,j}^s)$
16	$X_{i,j}^{s+1}=X_{i,j}^s+V_{i,j}^{s+1}$
17	else
18	$X_{i,j}^{s+1}=X_P^{s+1}+\left|X_{i,j}-X_P^{s+1}\right|\cdot A^{+}\cdot L$
19	Randomly generate vigilantes nv
20	for i in nv do
21	for j in m do
22	if fi > fg then
23	$X_{i,j}^{s+1}=B_{g,j}^s+\beta \cdot \left|X_{i,j}^s-B_{g,j}^s\right|$
24	else
25	$X_{i,j}^{s+1}=X_{i,j}^s+k\cdot (\frac{\left|X_{i,j}^s-X_w^s\right|}{(f_i-f_w)+\epsilon })$
26	for i in n do
27	Update fi
28	Update fg
29	if fg meet the requirement then
30	exit for
31	Return the optimal value

3.2. Fitness Function

In order to evaluate the algorithm objectively, we chose the average accuracy of a validation set as the training accuracy of the learning algorithm [40]. In order to facilitate the process and comparison of results, the error is defined as the fitness function value [41]. Equation (8) is the fitness function used for the evaluation of hyperparameters. A smaller fitness function value indicates better hyperparameters.

$$Fitness=1-\frac{{\displaystyle \sum _{i=1}^{n}accurac{y}_i}}{n}$$

(8)

4. Experiments

Experiments were conducted to verify the performance of HSSA by comparing it with several algorithms that are currently recognized as being excellent. A computer with an Intel i7 CPU, NVIDIA RTX3070 GPU, and 32 GB memory was used. The neural network model was built using TensorFlow. Random Search, Bayesian Optimization, CMA-ES, SA, GA, PSO, SSA, and HSSA were all written in Python 3. The details are as follows.

4.1. Convolutional Neural Network

The convolutional neural network (CNN) is a well-known deep learning architecture inspired by the biological vision mechanism [42]. Convolutional neural networks rely on convolution and pooling to identify information [43] and have been widely used in many fields such as target detection and image classification [44]. LetNet-5 [45] is a classic convolutional neural network and is often used to test algorithms. We used LetNet-5 in subsequent experiments to verify the performance of our method in low-dimensional space. A LetNet-5 model is shown in Figure 2.

In Figure 2, C1 and C2 are convolutional layers, P1 and P2 are pooling layers, and F1 and F2 are fully connected layers. In order to verify HSSA in complex networks, we propose a more complex convolutional neural network based on AlexNet [46], as shown in Figure 3.

In Figure 3, C1–C5 are convolutional layers, P1–P3 are pooling layers, and F1–F4 are fully connected layers. The role of each layer in the neural network is as follows.

• The input layer determines the type and style of the data entered.
• The convolutional layer is performed on two matrices. The convolution kernel moves on the input matrix with a certain step length. The output matrix is obtained after the convolution operation.
• The pooling layer is used for down sampling. The pooling layer continuously reduces the size of the data space. The number of parameters and calculations are decreased to control data over-fitting.
• Fully connected layers connect to all nodes of the previous layer, thus integrating extracted features in mapping distributed features to the sample label space.
• The output layer outputs the final result.

4.2. Performance Verification

The performance of HSSA was verified using two experiments with MNIST and Five Flowers datasets. The MNIST dataset is relatively simple and has a high classification accuracy. The Five Flowers dataset is relatively complex, and it is difficult to achieve a high classification accuracy using it. For the search effect in low-dimensional and high-dimensional spaces, two neural network models with different levels of complexity were used.

The experiment with the MNIST dataset uses the LetNet-5 neural network model, which is relatively simple. The optimized number of hyperparameters is small. As an optimization process of HSSA in low-dimensional space, the experimental process is relatively simple. The experiment on the Five Flowers dataset uses a deep convolutional neural network with a complex structure and more parameters. As an optimization process of HSSA in high-dimensional space, the experimental process is relatively complex. The experimental results were used to compare HSSA with Random Search, Bayesian Optimization, CMA-ES, SA, and GA, which are all recognized as excellent methods in the field.

The settings of all algorithms are shown in

Table 1. Settings of all algorithms.

Method	Setup
Random search	Completely random
Bayesian Optimization	Tree Parzen Estimator Gaussian process EI function
CMA-ES	Initial step: σ(0) = 0.618 (ub-lb) Initial evolutionary path: pσ(0) = 0, pc(0) = 0 Initial covariance matrix: C = I
SA	Initial temperature: T0 = 100 Descent rate: α = 0.99
GA	Variation rate: Pm = 0.2 Roulette wheel selection
PSO	Inertia weight: ω = 0.6 Learning factors: c1 = 2, c2 = 2
SSA	Discoverer ratio: 20% Detective ratio: 10% Alert threshold: 0.8
HSSA	Discoverer ratio: 20% Detective ratio: 10% Alert threshold: 0.8 Inertia weight: ω = 0.6 Learning factors: c1 = 2, c2 = 2

.

4.2.1. Experiment on MNIST Dataset

MNIST [47,48] is a classic dataset used for simple classification problems in machine learning. The dataset consists of 70,000 handwritten digital grayscale images, which are divided into 10 categories including numbers 0–9. These 70,000 images are divided into training and verification sets. Some sample pictures in the MNIST dataset are shown in Figure 4.

Lecun evaluated hyperparameters in very high dimensions and found that their performance changes were only attributed to a few hyperparameters [45]. We selected six important hyperparameters of the LetNet-5 convolutional neural network as the optimization objects, including the number of F1 units, number of F2 units, L2 weight decay, batch size, learning rate, and dropout rate. The range of each hyperparameter is listed in

Table 2. Range of hyperparameters to be optimized on MNIST.

Name	Range
Number of F1 units	128–1024
Number of F2 units	128–1024
L2 weight decay	0.0001–0.01
Batch size	16–128
Learning rate	0.0001–0.01
Dropout rate	0.1–0.5

.

Table 3. Parameter settings of the LetNet-5 convolutional neural network that remain unchanged.

Name	Value
Epochs	10
Input	Shape: 28 × 28; Dimensions: 1
Convolution layer 1	Size: 5 × 5; Strides: 1
Pooling layer 1	Size: 2 × 2; Strides: 2
Convolution layer 2	Size: 5 × 5; Strides: 1
Pooling layer 2	Size: 2 × 2; Strides: 2
Activation function	Relu; Softmax

shows the parameter settings of the LetNet-5 convolutional neural network that remain unchanged during the simple network experiment.

A criterion of evaluating the algorithm performance in experiments is the verification error generated during the training when the training time is fixed [49]. Since the running time in the experiment is mainly consumed by neural network training, the running time of the optimization algorithm itself can be negligible. Let n be the run time of the neural network, where n = iterations × individuals. The complexity of the algorithm is O(n). Moreover, different algorithms require different training times for each iteration, so it is unreasonable to use only the number of iterations to represent the training time. Therefore, in order to ensure that the training time of each algorithm is the same, each algorithm during the experiment was set according to

Table 4. Settings of each algorithm in the performance verification experiment on MNIST.

Method	Setup
Random search	700 iterations
Bayesian Optimization	700 iterations
CMA-ES	700 iterations
SA	700 iterations
GA	50 initial individuals; 700 generations
HSSA	10 individuals per generation; 70 generations

.

The error of each algorithm changes with the number of iterations is shown in Figure 5.

The comparison shows that HSSA is more effective than the other five algorithms at 280 iterations, and achieves the best effect at 530 iterations. HSSA has a strong global search capability of finding better hyperparameters than the other algorithms in simple neural networks. The experiment proves that HSSA performs well in a simple neural network.

4.2.2. Experiment on Five Flowers Dataset

The Five Flowers dataset (https://www.kaggle.com/alxmamaev/flowers-recognition, accessed on 3 April 2021) is a classic dataset used for complex classification problems in the machine learning field. The dataset consists of 3670 RGB images in 5 categories, including daisies, dandelions, roses, sunflowers, and tulips. These images are divided into a training set and validation set. Some sample pictures in the five flowers dataset are shown in Figure 6.

In this experiment, 11 important hyperparameters of complex convolutional neural networks were selected as optimization objects, including the number of F1 units, number of F2 units, number of F3 units, number of F4 units, L2 weight decay, batch size, learning rate, F1 dropout rate, F2 dropout rate, F3 dropout rate, and F4 dropout rate. The range of each hyperparameter is shown in

Table 5. The range of hyperparameters to be optimized on Five Flowers.

Name	Range
Number of F1 units	128–1024
Number of F2 units	128–1024
Number of F3 units	128–1024
Number of F4 units	128–1024
L2 weight decay	0.0001–0.01
Batch size	16–128
Learning rate	0.0001–0.01
F1 Dropout rate	0.1–0.5
F2 Dropout rate	0.1–0.5
F3 Dropout rate	0.1–0.5
F4 Dropout rate	0.1–0.5

.

Table 6. Parameter settings of the complex convolutional neural network that remain unchanged.

Name	Value
Epochs	20
Input	Shape: 32 × 32; Dimension: 3
Convolution layer 1	Size: 3 × 3; Strides: 2
Pooling layer 1	Size: 2 × 2; Strides: 2
Convolution layer 2	Size: 3 × 3; Strides: 2
Pooling layer 2	Size: 2 × 2; Strides: 2
Convolution layer 3	Size: 3 × 3; Strides: 1
Convolution layer 4	Size: 3 × 3; Strides: 1
Convolution layer 5	Size: 3 × 3; Strides: 1
Pooling layer 3	Size: 2 × 2; Strides: 2
Activation function	Relu; Softmax

shows parameter settings of the complex convolutional neural network that remain unchanged during experiment 2.

Settings of each algorithm in the experiment are shown in

Table 7. Settings of each algorithm in the experiment on Five Flowers.

Method	Setup
Random search	700 iterations
Bayesian Optimization	700 iterations
CMA-ES	700 iterations
SA	700 iterations
GA	50 initial individuals; 700 generations
HSSA	10 individuals per generation; 70 generations

.

The error of each algorithm varies with the number of iterations, as shown in Figure 7. Figure 7 shows that the performance of SA in the complex convolutional neural network model is unexpectedly poor. In order to test whether this situation is caused by errors in the experimental operation, the same experiment was carried out twice according to the same standard. Finally, we found that this situation is not caused by experimental operation errors, but by defects of SA itself. SA tends to fall into a local optimal solution and continues to oscillate around it during the search process. This shows that SA is not appropriate in high-dimensional spaces. Errors of the model optimized by the other four methods are all around 0.3. However, HSSA is still the best. Compared with the other excellent algorithms, HSSA shows a better global search capability and does not easily fall into local optimal solutions. HSSA can find better hyperparameters than the other algorithms in complex neural networks. The experiment proves that HSSA performs well in a complex neural network.

4.3. Meaning Verification

The meaning verification experiment compares the performance of HSSA, SSA, and PSO in neural networks, and analyzes the improvement. Although results of the performance verification experiment show that HSSA is superior to several other excellent algorithms, it does not prove meaningful improvements. In order to further verify the significance of the algorithm improvement, the meaning verification experiment was carried out.

The meaning verification experiment used the same neural networks model and datasets. We selected the original SSA and PSO as the comparison objects of HSSA. Algorithm settings during the experiment are shown in

Table 8. Settings of each algorithm in the meaning verification experiment.

Method	Setup
SSA	10 individuals per generation; 70 generations
PSO	10 individuals per generation; 70 generations
HSSA	10 individuals per generation; 70 generations

.

Results of the meaning verification experiment on the MNIST dataset are shown in Figure 8.

Figure 8 shows that the effect of HSSA surpasses that of the other two algorithms at 280 iterations, and the best effect is achieved at 530 iterations. Results of the meaning verification experiment on the Five Flowers dataset are shown in Figure 9.

Figure 9 shows that effects of SSA and PSO are almost the same. HSSA is more effective than the other two algorithms at 80 iterations and achieves the best effect at 470 iterations.

According to the experimental results for both MNIST and Five Flowers datasets, HSSA is better than SSA and PSO. It is proved that the optimization performance of the algorithm is greatly improved in both low-dimensional and high-dimensional spaces. It can be concluded that the improvement is of great significance.

4.4. Result Analysis

4.4.1. Optimization Effect Analysis

In order to further compare the eight methods used in experiments, experimental results are analyzed in

Table 9. Results analysis on MNIST.

Method	Mean Error	Minimum Error	Number of Iterations
Random search	0.0119	0.0115	277
Bayesian Optimization	0.0100	0.0097	62
CMA-ES	0.0107	0.0102	612
SA	0.0099	0.0096	68
GA	0.0109	0.0107	404
PSO	0.0102	0.0098	330
SSA	0.0106	0.0104	230
HSSA	0.0097	0.0086	530

and

Table 10. Results analysis on Five Flowers.

Method	Mean Error	Minimum Error	Number of Iterations
Random search	0.3537	0.3148	312
Bayesian Optimization	0.2837	0.2692	371
CMA-ES	0.2895	0.2830	580
SA	0.7555	0.7555	1
GA	0.3250	0.2869	570
PSO	0.3113	0.2973	90
SSA	0.3063	0.2957	190
HSSA	0.2714	0.2473	460

for the mean error, the minimum error, and the number of iterations required to reach the minimum error. The minimum error represents the ability of the algorithm to search for the optimal solution. A small minimum error indicates strong global search capability. The mean error represents the overall effect of the algorithm in the iterative process. It does not change dramatically for several generations. The number of iterations represents the speed of convergence. A small number of iterations indicates fast convergence.

From Table 9, it is observed that, except for the random search, the mean error of each algorithm is concentrated around 0.01, indicating that the overall effect of each algorithm is not significantly different. The minimum error of HSSA is more than 10.42% lower than that of the other classic algorithms, indicating that HSSA has a strong optimization performance. Compared with PSO and SSA, the minimum error of HSSA is reduced by 12.24% and 17.31%, indicating that our improvement is significant.

From Table 10, as SA tends to fall into a local optimal solution in high-dimensional problems, it is not suitable for complex neural networks. The mean error of HSSA is at least 4.34% lower than that of the other algorithms, which shows that HSSA has the best overall optimization result. The minimum error of HSSA is at least 8.16% lower than that of the other classic algorithms, indicating that the optimization of HSSA is excellent. The minimum error of HSSA is 16.82% and 16.37% lower than that of PSO and SSA, respectively, indicating that our improvement is significant.

In summary, HSSA has an excellent performance in both low-dimensional and high-dimensional spaces. Its performance has been greatly improved.

4.4.2. Global Search Capability Analysis

Although previous experiments prove the excellent performance of HSSA, they cannot prove the global optimization solution of the improved HSSA.

In order to verify the global search capability of the improved HSSA, further analysis was conducted, as shown in

Table 11. Positions of individuals of PSO in the experiment on MNIST.

	Bs	L2	F1	F2	Dr	Lr
1	127	0.0032	716	340	0.10	0.0008
2	127	0.0032	716	340	0.10	0.0008
3	127	0.0032	716	340	0.10	0.0008
4	127	0.0032	716	340	0.10	0.0008
5	127	0.0032	716	340	0.10	0.0008
6	127	0.0032	716	340	0.10	0.0008
7	127	0.0032	716	340	0.10	0.0008
8	127	0.0032	716	340	0.10	0.0008
9	128	0.0017	521	409	0.17	0.0029
10	98	0.0035	1003	507	0.24	0.0014

,

Table 12. Positions of individuals of SSA in the experiment on MNIST.

	Bs	L2	F1	F2	Dr	Lr
1	128	0.0100	1024	1024	0.50	0.0100
2	107	0.0084	954	973	0.41	0.0094
3	99	0.0097	1017	912	0.47	0.0086
4	68	0.0006	579	1015	0.15	0.0059
5	97	0.0029	629	283	0.49	0.0031
6	85	0.0061	358	420	0.30	0.0089
7	61	0.0065	690	421	0.36	0.0099
8	51	0.0074	466	212	0.34	0.0011
9	91	0.0066	306	677	0.29	0.0055
10	27	0.0081	447	379	0.35	0.0004

,

Table 13. Positions of individuals of HSSA in the experiment on MNIST.

	Bs	L2	F1	F2	Dr	Lr
1	16	0.0001	128	128	0.10	0.0001
2	21	0.0001	132	130	0.11	0.0001
3	19	0.0008	146	183	0.13	0.0001
4	17	0.0077	838	959	0.16	0.0066
5	72	0.0020	310	914	0.34	0.0089
6	123	0.0066	436	498	0.28	0.0041
7	45	0.0036	451	180	0.23	0.0003
8	111	0.0037	314	924	0.14	0.0024
9	116	0.0058	1019	658	0.45	0.0009
10	42	0.0046	957	614	0.43	0.0024

,

Table 14. Positions of individuals of PSO in the experiment on Five Flowers.

	Bs	L2	F1	F2	F3	F4	Dr1	Dr2	Dr3	Dr4	Lr
1	68	0.0010	544	137	241	232	0.50	0.50	0.50	0.50	0.0010
2	68	0.0010	544	137	241	232	0.50	0.50	0.50	0.50	0.0010
3	68	0.0010	544	137	241	232	0.50	0.50	0.50	0.50	0.0010
4	68	0.0010	544	137	241	232	0.50	0.50	0.50	0.50	0.0010
5	68	0.0010	544	137	241	232	0.50	0.50	0.50	0.50	0.0010
6	68	0.0010	544	137	241	232	0.50	0.50	0.50	0.50	0.0010
7	68	0.0010	544	137	241	232	0.50	0.50	0.50	0.50	0.0010
8	83	0.0010	472	189	380	176	0.45	0.43	0.49	0.48	0.0010
9	49	0.0038	718	444	426	264	0.34	0.40	0.37	0.39	0.0007
10	72	0.0027	385	235	192	370	0.25	0.38	0.32	0.47	0.0015

,

Table 15. Positions of individuals of SSA in the experiment on Five Flowers.

	Bs	L2	F1	F2	F3	F4	Dr1	Dr2	Dr3	Dr4	Lr
1	61	0.0012	128	184	184	184	0.48	0.47	0.50	0.47	0.0014
2	119	0.0001	153	455	616	510	0.26	0.40	0.35	0.25	0.0022
3	78	0.0008	997	208	482	1010	0.39	0.23	0.20	0.16	0.0002
4	61	0.0015	630	798	176	703	0.47	0.48	0.47	0.46	0.0090
5	66	0.0048	832	745	389	446	0.48	0.43	0.18	0.28	0.0046
6	67	0.0060	432	944	593	631	0.27	0.37	0.22	0.24	0.0002
7	64	0.0097	771	199	326	454	0.43	0.10	0.14	0.32	0.0093
8	21	0.0020	598	480	466	830	0.43	0.17	0.49	0.49	0.0030
9	51	0.0005	426	737	957	870	0.36	0.25	0.31	0.15	0.0094
10	43	0.0057	610	722	810	908	0.49	0.45	0.11	0.27	0.0008

and

Table 16. Positions of individuals of the HSSA in the experiment on Five Flowers.

	Bs	L2	F1	F2	F3	F4	Dr1	Dr2	Dr3	Dr4	Lr
1	128	0.0001	572	497	555	222	0.41	0.41	0.41	0.41	0.0010
2	35	0.0021	354	500	573	193	0.43	0.30	0.33	0.35	0.0080
3	106	0.0032	181	326	481	655	0.23	0.26	0.48	0.13	0.0034
4	55	0.0040	138	714	244	265	0.13	0.42	0.31	0.13	0.0059
5	28	0.0059	294	858	276	150	0.22	0.34	0.17	0.14	0.0007
6	63	0.0047	856	520	211	285	0.49	0.17	0.47	0.27	0.0008
7	23	0.0096	241	575	647	966	0.39	0.37	0.21	0.19	0.0026
8	43	0.0070	715	410	489	378	0.49	0.14	0.24	0.45	0.0068
9	74	0.0031	263	618	652	158	0.34	0.30	0.40	0.25	0.0081
10	28	0.0028	922	505	162	617	0.34	0.48	0.15	0.12	0.0028

, which indicate the location of each individual in the population at the end of the algorithm operation. Small differences between individuals in the population indicate that the algorithm can easily fall into a local optimal solution. On the contrary, large differences between individuals indicate that the population contains more global information due to the global search ability [39]. In these tables, Bs is the batch size, L2 is L2 weight decay, F1–F4 is the number of F1–F4 units, Dr is the dropout rate, and Lr is the learning rate.

In order to visually compare the degree of dispersion of each group of data, their standard deviations were used as evaluation criteria. A large standard deviation indicates a high degree of dispersion [50]. Because these data are multi-dimensional, they can be regarded as discrete points in a high-dimensional space, and the difference between the data is the spatial distance between the points. For n points in the m-dimensional space, the standard deviation can be calculated using the following formulas:

$$\left|x_a-x_b\right|=\sqrt{{\displaystyle \sum _{i=1}^n{(x_a^i-x_b^i)}^2}}$$

(9)

$$\overline{x}=(\frac{1}{n}{\displaystyle \sum _{i=1}^n{x}_i^1},\frac{1}{n}{\displaystyle \sum _{i=1}^n{x}_i^2},\cdots ,\frac{1}{n}{\displaystyle \sum _{i=1}^n{x}_i^m})$$

(10)

$$S=\sqrt{\frac{1}{n}{\displaystyle \sum _{i=1}^n{(\left|x_i-\overline{x}\right|)}^2}}$$

(11)

In addition, due to the huge difference in the value range of each parameter, using original values of parameters will cause different parameters to have different degrees of influence on the standard deviation. Therefore, each parameter needs to be processed, so the original values of each parameter were converted to its relative position within the value range [51]. The processing method is shown in Equation (12):

$$t^{\prime }=\frac{t-t_{\min }}{t_{\max }-t_{\min }}$$

(12)

where t is the original value of the parameter, and t′ is the relative position of the parameter. The converted relative positions are not affected by the value range. Each relative position has the same influence on the standard deviation.

Figure 10 shows the standard deviations when the iteration of PSO, SSA, and HSSA is completed in experiments for two different datasets.

Figure 10 shows the standard deviations of PSO, SSA, and HSSA on the MNIST dataset and Five Flowers dataset. It can be observed that regardless of experiments on the MNIST dataset or Five Flowers dataset, the standard deviation of PSO is very small. It means that PSO can easily fall into a local optimal solution as assumed. In experiments of the MNIST dataset and Five Flowers dataset, the standard deviation of HSSA is larger than that of PSO, by 0.5574 and 0.5323, respectively. In experiments of the MNIST dataset, the standard deviation of HSSA is 0.1174 higher than that of SSA. This shows that individuals of HSSA are even more scattered than those of SSA in experiments with the MNIST dataset. In experiments with the Five Flowers dataset, the standard deviation of HSSA is 0.085 lower than that of SSA, which is acceptable. In summary, the global search ability of HSSA could be verified by the experiment results.

4.5. Stability Analysis of HSSA

In the experiments, initial values of hyperparameters in optimization algorithms were randomly generated. Different initial values may lead to different experimental results. In order to verify the optimization ability of HSSA in the neural network under randomness, we conducted five experiments on MNIST and Five Flowers, respectively. The experiment settings were the same for both experiments. The mean and standard deviation of experimental results represent the effect of randomness on the HSSA performance.

Table 17. Mean and standard deviation of 5 experiments on MNIST.

Experiment	Mean Error	Minimum Error
Experiment 1	0.0095	0.0089
Experiment 2	0.0104	0.0092
Experiment 3	0.0094	0.0085
Experiment 4	0.0099	0.0081
Experiment 5	0.0108	0.0093
Mean Value	0.0100	0.0088
Standard Deviation	0.00053	0.00048

and

Table 18. Mean and standard deviation of 5 experiments on Five Flowers.

Experiment	Mean Error	Minimum Error
Experiment 1	0.2697	0.2457
Experiment 2	0.2821	0.2472
Experiment 3	0.2749	0.2464
Experiment 4	0.2701	0.2403
Experiment 5	0.2663	0.2396
Mean Value	0.2726	0.2438
Standard Deviation	0.00548	0.00322

show the mean and standard deviation of the experimental results of HSSA in LetNet-5 and complex networks. Table 17 shows that the LetNet-5 model optimized by HSSA obtains a low error on the MNIST dataset. In addition, both average and minimum errors are very low. Standard deviations of the average and minimum errors are also very low. This shows that results of the five experiments are all very satisfactory, and the optimization performance of HSSA in a simple convolutional neural network model is very stable. Table 18 shows that the complex neural network optimized by HSSA has a gap between the mean and minimum errors on the Five Flowers dataset. The main reason for this phenomenon is that Five Flowers is a complex dataset, which is different from MNIST. A high classification accuracy cannot be achieved in the initial and middle stage optimization. Therefore, the final mean is high due to the high error in the initial and middle optimization stage. Even so, standard deviations of mean errors and minimum errors in the complex neural network are within an acceptable range. This shows that the optimization performance of HSSA in the complex convolutional neural network model is relatively stable.

Comparing Table 17 and Table 18 with Table 9 and Table 10, respectively, it can be seen that the minimum error of each experiment of HSSA is smaller than that of the other algorithms. This shows that the optimization performance of HSSA is not affected by randomness and verifies the stability of HSSA.

5. Discussion

Bayesian optimization, random search, and grid search are classic methods in neural network hyperparameter optimization. Although optimization methods based on heuristic algorithms and other methods proposed in recent years have also been used in the hyperparameter optimization of neural networks, they generally have poor global search capabilities and easily fall into local optimal solutions. Most of them are only suitable for solving a specific problem without universality. Therefore, the current heuristic algorithms are not effective for hyperparameter optimization. This paper improves heuristic algorithms for the hyperparameter optimization of neural networks. It provides a new research direction for solving problems of neural network hyperparameter optimization, which is to study new heuristic algorithms and apply them in the hyperparameter optimization.

The research in this paper validates the hybrid heuristic algorithm. The combination of different algorithms results in a new method that can benefit from the advantages of each algorithm. As a hybrid heuristic algorithm, HSSA embodies advantages of both PSO and SSA.

However, HSSA outperforms other methods when it iterates 280 times in the simple network. Although the optimal solution of HSSA is better than that of other methods, it requires sufficient iterations. In addition, HSSA contains some parameters that need to be manually set. If these parameters are not set properly, the algorithm may not be as effective as some traditional methods. This aspect also needs to be studied through a large number of experiments in future work.

HSSA is essentially an optimization method. It can also be used to solve other optimization problems. Moreover, as a novel heuristic algorithm, HSSA is a kind of swarm intelligence algorithm with potential parallelism, which has not been fully utilized. Due to the lack of study of the SSA, its true value has not yet been fully discovered. In future work we will further improve the SSA to take full advantage of its global search capability.

6. Conclusions

In order to improve neural network hyperparameter optimization for global search ability, we propose a new HSSA algorithm to avoid local optimal solutions. This algorithm fixes the data overflow defect of the original SSA and combines the advantages of the strong ability of SSA to find a global optimization solution with the high search speed of PSO. It performs well in both simple and complex networks.

The performance verification experiments on simple and complex networks prove that HSSA is an excellent hyperparameter optimization method. The minimum error of HSSA in simple networks is about 10% lower than that of the other classic algorithms. It is about 8% lower than that of the other classic algorithms in complex networks. Experimental results show that HSSA has excellent optimization performance to find better solutions than other algorithms.

Results of the meaning verification experiment on simple and complex networks prove the significance of HSSA’s improvement. By combining SSA and PSO, the data overflow problem caused by SSA is solved and the search speed is improved. The performance of the algorithm in neural networks is improved. In a simple network, the minimum error of HSSA is about 12% and 17% lower than that of PSO and SSA, respectively. In a complex network, the minimum error of HSSA is about 12% lower than that of PSO and SSA.

In addition, the stability analysis of HSSA proves that the optimization performance of HSSA is not affected by randomness, and shows that HSSA is stable and adaptable. In short, based on the research results of this article, HSSA is proven to be an excellent hyperparameter optimization algorithm.

However, this study has limitations. During the optimization process, the neural network may run hundreds or thousands of times. It takes hours to process a large dataset for some of the new network models proposed in recent years, which thus have a high time cost. Therefore, it is difficult to experiment with overly complex cases. In the future, the optimization algorithm will be applied to more complex situations with the continuous improvement in computer performance.

Although some achievements were made in this study, some future work needs to be undertaken. Firstly, in simple networks, the effect of HSSA is not significant when the number of iterations is small. Speeding up the early search speed will be a future research direction. Secondly, some parameters have an impact on the optimization performance of HSSA. Another future research direction will be to study setting appropriate parameters to improve the optimization performance. Thirdly, the combination of two algorithms provides a direction for new algorithms possessing the advantages of these two algorithms. Finally, limited by computer performance, current hyperparameter optimization algorithms are not suitable for particularly complex situations. Simplifying neural networks or improving computational efficiency will be a future research focus.