Hybrid Particle Swarm Optimization Algorithm Based on the Theory of Reinforcement Learning in Psychology

Abstract

To more effectively solve the complex optimization problems that exist in nonlinear, high-dimensional, large-sample and complex systems, many intelligent optimization methods have been proposed. Among these algorithms, the particle swarm optimization (PSO) algorithm has attracted scholars’ attention. However, the traditional PSO can easily become an individual optimal solution, leading to the transition of the optimization process from global exploration to local development. To solve this problem, in this paper, we propose a Hybrid Reinforcement Learning Particle Swarm Algorithm (HRLPSO) based on the theory of reinforcement learning in psychology. First, we used the reinforcement learning strategy to optimize the initial population in the population initialization stage; then, chaotic adaptive weights and adaptive learning factors were used to balance the global exploration and local development process, and the individual optimal solution and the global optimal solution were obtained using dimension learning. Finally, the improved reinforcement learning strategy and mutation strategy were applied to the traditional PSO to improve the quality of the individual optimal solution and the global optimal solution. The HRLPSO algorithm was tested by optimizing the solution of 12 benchmarks as well as the CEC2013 test suite, and the results show it can balance the individual learning ability and social learning ability, verifying its effectiveness.

1. Introduction

In order to more effectively solve problems in many fields in real life, scholars mathematically model them; that is, they establish an optimization model [1]. In the process of this mathematical modeling, it is found that some problems are difficult to accurately model or solve. To facilitate the solution of traditional methods, the target usually needs to be processed, which increases the complexity of the problem [2]. The intelligent optimization method does not have this limitation and can solve the target model more conveniently [3,4]. Li et al. [5], Dokeroglu et al. [6] and Xue et al. [7] presented some comprehensive surveys of the state-of-the-art schemes on intelligent optimization for feature selection, which is helpful for optimization performance. Therefore, intelligent optimization methods have developed rapidly. Intelligent optimization methods include the genetic algorithm (GA) [8], the artificial bee colony algorithm (ABC) [9], the simulated annealing algorithm (SA) [10], the particle swarm optimization (PSO) algorithm [11], etc. Among them, PSO has attracted scholars’ attention because of its simple structure and easy implementation [12].

PSO was first proposed by Kennedy and Eberhart [12]. The initially proposed optimization effect of PSO was unexceptional. Later, scholars usually attempted to improve the inertia weight ω with a nonfixed value in the PSO, and the particle renewal formula was first proposed by Yuhui and Eberhart [13]. Subsequent scholars have carried out a lot of research regarding how the optimization ability of PSO can be improved. PSO usually randomly generates various potential solutions in the range of the solution of the optimization problem, which are called “particles”. In reference [14], in order to improve the quality of initial particles, Tian et al. replaced the method of generating initial particles via random mapping in PSO with logical mapping. Chen et al. [15] first used random mapping to generate initial particles and then combined this method with a reinforcement learning strategy [16] to generate another batch of reinforcement particles. In this method, after comparing the fitness values of the particles generated using the two methods, the particles with good fitness values are left as the initial particles. Gao et al. [17] first initialized particles via sinusoidal mapping and then used a reinforcement learning strategy to generate a batch of reinforcement particles. Then, they compared the advantages and disadvantages of the two batches of particles to leave particles closer to the optimal solution. In this method, new particles are generated through the two cores of PSO’s updated velocity and displacement formula. In the velocity formula, the degree to which the velocity of the new particle is affected by the previous velocity is determined by the inertia weight ω. The degree of influence of the global optimal solution and the individual optimal solution is controlled by the acceleration coefficients c1 and c2. Therefore, ω and c1/c2 have great influence on the final optimization results. To this end, the strategies used to improve ω include the linear strategy [13], the nonlinear strategy [18], the fuzzy rule [19], the chaotic strategy [15], etc. With regard to the acceleration coefficient, sometimes variable acceleration coefficients [20], fixed value acceleration coefficients [21], etc., are used. However, some scholars have improved other values of the updated formula. For example, Xu et al. proposed a dimension learning strategy to improve the individual optimal solution. In this method, the value of each dimension of each individual optimal solution is replaced by the value of each dimension of the global optimal solution one by one. If the effect is positive, the value of the corresponding dimension of the global optimal solution will be retained, and if not, the original state will be maintained [3]. Liang et al. proposed a comprehensive learning strategy to remove the social learning aspect from the speed update formula of classical PSO so that all the remaining individual optimal solutions have the opportunity to learn from the historical individual optimal solutions of other particles, which creates the opportunity for particles to learn from all of the individual optimal solutions [22]. Li et al. combined the improvement of the comprehensive learning strategy and mutation strategy to improve the optimization ability of PSO [23]. Mendes et al. established a speed update strategy in which the particle speed update depends not only on the historical optimal solution of the particle, but also on the historical optimal solution of all other particles [24]. Some scholars applied a mutation strategy to the position of particles to make particles jump out of the local optimal solution. After updating the historical individual optimal particles and historical global optimal particles in PSO, Wang et al. used a mutation strategy to mutate them [25]. This mutation strategy includes Cauchy, Levy, and Gaussian mutations; then, a roulette selection mechanism is used to select mutation factors [26]. Li et al. performed a mutation operation on the global optimal solution in the algorithm when improving PSO. The mutation factor was generated from the difference between two random particles in the population [23]. This research represents the main improvements made by scholars regarding the ontology of the PSO algorithm, while some scholars have combined other algorithms with PSO to form a better algorithm. For example, in reference [27], PSO and GSA were combined to form a hybrid algorithm. The aim was to combine the local development performance of the GSA and the global exploration performance of PSO to form a complementary algorithm. PSO can also be mixed with the sine cosine algorithm [28], the genetic algorithm [29], etc. However, these modified PSOs are still likely be categorized as individual optimal solutions, leading to the transition of the optimization process from global exploration to local development.

In summary, the main challenges of the PSO algorithm are to improve the optimization ability of both the local exploitation and the global exploration by combining all kinds of other algorithms. This leads to the transition of the optimization process from global exploration to local development.

To improve the optimization performance, in this paper, we propose a Hybrid Reinforcement Learning Particle Swarm Algorithm (HRLPSO) based on the theory of reinforcement learning in psychology, which is based on teamwork and runs parallelly.

(1)

A Hybrid Reinforcement Learning Particle Swarm Algorithm was proposed. To enhance the optimization capability of HRLPSO, five strategies were applied to improve the traditional PSO in this work. (i) An opposition-based learning strategy was combined with random mapping to generate the initial population; (ii) cubic mapping and adaptive strategies were combined and applied to the weights; (iii) the c_i parameter was controlled to vary nonlinearly within a certain range; (iv) a dimensional learning strategy was applied to the optimal solution; (v) Cauchy and Gaussian mutation strategies were used in the optimal solution to increase the diversity of the solutions.

(2)

The results regarding standard functions show that the proposed HRLPSO strategy works well in both stand-alone and ensemble applications, and the results regarding the CEC2013 test suite further demonstrate the good optimization capability of HRLPSO.

(3)

Compared with the existing schemes, the main contributions of the proposed HRLPSO are as follows: (i) The theory of reinforcement learning in psychology is firstly applied and the opposition-based learning strategy is proposed to generate the initial population of the PSO. (ii) Unlike the traditional PSO algorithm, which only uses a few hybrid methods, the proposed HRLPSO fully considers the improvement measures at each stage and the five hybrid methods stated above in (1) are applied to improve the optimization performance.

2. Particle Swarm Optimization Algorithm

The particle swarm optimization algorithm is an evolutionary algorithm. The algorithm first generates a set of “solutions” within the approximate range of the solution of the optimization problem, that is, “particles” X_i = (x_i1, x_i2, …, x_iD). The value of i is an integer from 1 to N, N is the number of particles, and D is the dimension of particles. Then, by comparing the corresponding objective function values of these particles in the optimization problem, the historical individual optimal solution Pbest_i = (pbest_i1, pbest_i2, …, pbest_iD) and the historical global optimal solution Gbest = (gbest₁, gbest₂, …, gbest_D) are obtained. The new particles are updated using the following formula:

$$v_{(i+1)d}=\omega \ast v_{id}+c_{1}rand()(pbes{t}_{id}-x_{id})+c_{2}rand()(gbes{t}_d-x_{id}),$$

(1)

$$x_{(i+1)d}=x_{id}+v_{id},$$

(2)

In Equation (1), v represents the velocity of particles, and all the velocity vectors are represented by V_i = (v_i1, v_i2, …, v_iD). The values of c₁ and c₂ are weight factors that control particles’ individual learning and social learning, and ω is the inertia weight that controls the influence of the previous particle velocity on the updated particle velocity.

3. Hybrid Reinforcement Learning Particle Swarm Optimization Algorithm

3.1. Initial Population Based on Positive Reinforcement Learning

The initial population reinforcement theory based on positive reinforcement learning is a theory proposed by Skinner, an American psychologist and behavioral scientist. Skinner was one of the founders of new behaviorist psychology. He believed that people or animals will display certain behaviors to act on the environment in order to achieve a certain purpose. When the consequences of such behavior are beneficial to the individual, such behavior will repeat in the future; when unfavorable, this behavior weakens or disappears. People can use this method of positive reinforcement or negative reinforcement to change the consequences of behavior and modify their behavior. This is reinforcement theory, also known as behavior modification theory [5]. The convergence speed and accuracy of the particle swarm optimization algorithm are easily affected by the quality of the initial population. In order to improve the quality of the initial population, reinforcement learning is applied to the process of initializing the population.

In the optimization process of various algorithms, some random individuals are randomly generated in the range of solutions as potential solutions and then continuously approach the optimal solution through various iterative mechanisms to produce the optimal solution. However, these algorithms can be improved by later scholars so as to make the algorithm approach better and faster and produce the optimal solution. In this study, reinforcement learning was applied to the algorithm. Reinforcement learning [12] is defined as follows:

Suppose a real number x^rn ∈ [A, B], and the opposite number of x^on is defined as follows:

$$x^{on}=A+B-x^{rn}$$

(3)

The remaining two definitions are based on the definitions above. Apply the definitions above to the position of an algorithm, such as the PSO algorithm, in which particle X_i^rn = (x_i1^rn, x_i2^rn, …, x_iD^rn), and enhanced particle X_i^on = (x_i1^on, x_i2^on, …, x_iD^on), where x^rn ∈ [A, B], and

$$x_{ij}^{on}=A_i+B_i-x_{ij}^{rn}$$

(4)

Then, by comparing the fitness values of X_i^rn and X_i^on in the optimized objective function f(x), the particles with excellent fitness values are left.

3.2. Chaos Adaptive Inertia Weight

The optimization ability of PSO can be effectively improved by reasonably setting the change in the inertia weight coefficient. It has been proved in [9,26] that a linear decline in inertia weight within a certain range can effectively enhance the performance of PSO. The linear decline formula is

$$\omega ={\omega }_{\max }-\frac{\left({\omega }_{\max }-{\omega }_{\min }\right)}{\max gen}i,$$

(5)

where ω is the value of the inertia weight coefficient under the current number of iterations, ω_max/ω_min is the maximum/minimum value of the inertia weight coefficient, i is the current number of iterations, and maxgen is the maximum number of iterations. At present, the most commonly used ω_max/ω_min values in this formula are 0.9/0.4, respectively. In this study, cubic mapping was applied to linearly decreasing weight coefficients as follows [27]:

$$x_{n+1}=a{x}_n^3+\left(1-a\right)x_n,$$

(6)

where x_n denotes the n-th chaotic state in the range of [−1, 1]; the initial value x₀ of x_n cannot take 0; and a is the bifurcation coefficient in the semi-open interval (0, 4]. When the value of a increases from zero, the fixed points in Figure 1, the bifurcation graph generated by Equation (6), vary from 1 to 2 and, after that, from 4 to 2n. This variation presents as unlimited and stable, but when a increases close to 3.598076211, the duration proves to be infinite, even aperiodic. When a lies in the range of [3.598076211, 4], the chaotic state occurs and the system presents as unstable when a is bigger than 4, as depicted in Figure 1, where the different random numbers are displayed in different colors.

After setting the absolute value range of the mapped fluctuation (the range is obtained through continuous experimental parameter adjustment), the absolute value range of the fluctuation is as follows:

$$V(i)=Max-\frac{i}{\max gen}Max,$$

(7)

where V(i) represents the absolute value of the fluctuation of the mapping under the current number of iterations, and Max is the absolute value of the fluctuation at the first iteration. Combined with cubic mapping, a linearly decreasing mixed disturbance is formed. The combined formula is as follows:

$$C\left(i\right)=x\left(i\right)\ast V\left(i\right),$$

(8)

Finally, the chaotic adaptive inertia weight is obtained by adding it to Equation (5).

$$\omega \left(i\right)=\omega \left(i\right)+C\left(i\right),$$

(9)

The whole process is shown in Figure 2, where the variables are depicted as the blue curves

3.3. Adaptive Learning Factor

The research regarding learning factors usually focuses on two aspects. On the one hand, the learning factor can be set to a fixed constant. The most typical example of this is the original PSO algorithm. The value of both learning factors is set to 2 in the literature [1]. On the other hand, the learning factor can be set as an adaptive learning factor. Usually, the value of the learning factor is fixed in a certain range and changes with the number of iterations. In the typical research literature [11,16,28,29], this value increases or decreases linearly or nonlinearly between 0.5 and 2.5 as the number of iterations changes. This study is based on adaptive learning factors. The formula of the variable learning factor is as follows:

$$c_1\left(i\right)=\alpha \times \sqrt{1-{\left(1-\left(i/\max gen\right)\right)}^2}+\beta ,$$

(10)

$$c_2\left(i\right)=\alpha \times \left(1-\sqrt{1-{\left(1-\left(i/\max gen\right)\right)}^2}\right)+\beta ,$$

(11)

where α = 2, β = 0.5. The iterative curve of the learning factors is shown in Figure 3.

3.4. Update Strategy

3.4.1. Dimension Learning

Xu et al. proposed a dimension learning strategy. The principle of this strategy is to replace the value of each dimension of the historical individual optimal solution with the value of the corresponding dimension of the historical global optimal solution. If the objective function value of the optimization problem corresponding to the replaced historical individual optimal solution is better, the value of the replaced dimension will be retained [4]. The advantage of this work is that the best solution is selected from the historical individual optimal solution with the reinforcement learning strategy, and it is compared with the historical global optimal solution, improving the historical global optimal solution. The updated formula is as follows:

$$v_{(i+1)d}=\omega \ast v_{id}+c_{1}rand()(pbes{t}_{id}^{dl}-x_{id})+c_{2}rand()(gbes{t}_d^{dl}-x_{id}),$$

(12)

Pbest_i^dl and Gbest^dl represent the historical individual optimal solution and the historical global optimal solution of the reinforcement learning strategy, respectively.

3.4.2. Mutation

The PSO algorithm has inherent defects and can be easily categorized as local optimization. Particle mutation is an effective strategy to alleviate this situation. The random number of Gaussian distribution functions is mainly concentrated near 0, meaning Gaussian mutation is suitable for particle exploration. Compared with the Gaussian distribution function, the number randomly generated by the Cauchy distribution function is far from 0, meaning that the Cauchy mutation is suitable for particle development. In our work, we aimed to mutate the particle when the particle of the PSO algorithm was categorized as local optimization and simultaneously carried out Gaussian mutation and Cauchy mutation on the particle. The mutation method that produced better results was adopted. The mutation formula is as follows:

$$p_{id}^{dlm}=p_{id}^{dl}+mutatio{n}_d()$$

(13)

$$p_{gd}^{dlm}=p_{gd}^{dl}+mutatio{n}_d()$$

(14)

where mutation_d () is the mutation factor.

4. Experimental Setup

In this study, classical test functions were used to test the performance of the algorithm, including seven unimodal functions (F1–F7) and five multimodal functions (F8–F12) [30]. When comparing HRLPSO with the other four algorithms, the population size was set to 30, and each algorithm optimized the test function 20 times. When comparing the performance of the improved method separately, the number of iterations was 1000, and when comparing the performance of HRLPSO with the other four algorithms, the number of iterations was 10,000. The maximum speed limit was consistent with the range of the test function. In addition, in this study, the average value of the final result of the algorithm’s optimization of the test function after 20 times is displayed in bold for easy observation. CIPSO was directly applied to engineering problems in the original text, and the performance of the algorithm was analyzed based on the results of engineering problems. In CLPSO and DLPSO, standard test functions are mainly used to assess the performance of an algorithm. The parameter settings of all the algorithms in this work are shown in

Table 1. Parameter settings.

Algorithm	Parameter		Reference
PSO	the population size is 30, each algorithm is optimized 20 times, the number of iterations is 10,000, the maximal speed is within the range of F1~F12	w: 1, c1: 2, c2: 2	[8]
CIPSO		w: 0.9~0.4, c1: 3.5~0.5, c2: 0.5~3.5	[31]
CLPSO		w: 0.9~0.4, c: 1.5	[18]
DLPSO		w: 0.7298, c1: 1.5, c2: 0.5~2.5	[3]
HRLPSO		w: 0.9~0.6, c1: 2.5~0.5, c2: 0.5~2.5, a: 4, Max: 0.05	-

.

In the original literature, the capabilities of CIPSO were evaluated by optimizing the results for application to engineering problems. In the original literature, two algorithms, CLPSO and DLPSO, were mainly used to test the algorithm performance with standard test functions. The parameters of the PSO variants are shown in Table 1.

Figure 4 displays the flow chart of HDLPSO, in which Fit is the fitness value of the solution. As test functions exist in minimum values, the solution with the smaller fitness value was taken as the better solution when comparing the fitness values.

5. Discussion

5.1. Test Results of the PSO Variants under Benchmark

Taking 12 benchmark functions as experimental objects, we compared the optimization results of HRLPSO and four other algorithms. There were 10,000 iterations. The results of the comparison of HRLPSO and the four other algorithms are shown in

Table 2. Optimization results of HRLPSO and other algorithms under benchmark.

F	D	Item	PSO	CIPSO	CLPSO	DLPSO	HRLPSO
F1	30	Mean	2 × 103	1.72 × 101	4.43 × 10−9	0.00	0.00
		S.D.	5.23 × 103	8.42	2.53 × 10−9	0.00	0.00
		Rank	4	3	2	1	1
F2	30	Mean	1.50 × 101	5.86 × 10−1	8.66 × 10−8	3.49 × 10−43	0.00
		S.D.	8.89	3.02 × 10−1	3.54 × 10−8	1.53 × 10−48	0.00
		Rank	5	4	3	2	1
F3	30	Mean	1.15 × 104	4.96 × 102	8.40 × 103	1.07 × 103	0.00
		S.D.	1.08 × 104	2.25 × 102	8.92 × 103	2.35 × 103	0.00
		Rank	5	2	4	3	1
F4	30	Mean	0.00	4.35	1.28	2.01 × 10−13	0.00
		S.D.	0.00	1.06	5.87 × 10−1	8.98 × 10−13	0.00
		Rank	1	4	3	2	1
F5	30	Mean	5.70 × 101	6.31 × 102	2.31 × 102	4.91 × 101	1.99 × 10−1
		S.D.	1.26 × 102	4.24 × 102	6.75 × 102	4.01 × 101	8.91 × 10−1
		Rank	3	5	4	2	1
F6	30	Mean	1.01 × 103	1.80 × 101	6.38 × 10−9	0.00	0.00
		S.D.	3.11 × 103	6.69	4.53 × 10−9	0.00	0.00
		Rank	4	3	2	1	1
F7	30	Mean	1.34	1.19 × 10−2	9.17 × 10−4	2.16 × 10−2	3.49 × 10−4
		S.D.	3.54	5.20 × 10−3	3.78 × 10−4	1.46 × 10−2	3.13 × 10−4
		Rank	5	3	2	4	1
F8	30	Mean	6.51 × 101	6.20 × 101	7.06	5.42	0.00
		S.D.	4.26 × 101	1.54 × 101	2.58	3.88	0.00
		Rank	5	4	3	2	1
F9	30	Mean	7.24	3.11	1.90 × 101	1.25 × 10−1	8.88 × 10−16
		S.D.	8.44	4.73 × 10−1	4.70 × 10−1	3.85 × 10−1	0.00
		Rank	4	3	5	2	1
F10	30	Mean	9.02 × 101	1.11	1.85 × 10−10	1.41 × 10−2	0.00
		S.D.	2.78 × 101	3.84 × 10−2	1.54 × 10−1	2.24 × 10−2	0.00
		Rank	5	4	2	3	1
F11	30	Mean	1.49 × 10−1	7.92 × 10−1	4.23 × 10−11	3.63 × 10−2	1.57 × 10−32
		S.D.	3.87 × 10−2	3.83 × 10−1	2.78 × 10−11	5.07 × 10−2	2.81 × 10−48
		Rank	4	5	2	3	1
F12	30	Mean	2.07	1.93e	4.38 × 10−10	2.69 × 10−2	1.35 × 10−32
		S.D.	3.20	1.13	3.05 × 10−10	9.00 × 10−2	2.81 × 10−48
		Rank	5	4	2	3	1
Average Rank			4	3.89	2.78	2.44	1
Final Rank			5	4	3	2	1

. For the other four functions aside from the HRLPSO function in the table, the global optimal value of 0 could be obtained. It was shown that even the most original PSO could obtain the global optimal value of 0 on function F4. However, the global optimal results of HRLPSO on functions F1, F2, F3, F4, F6, F8, and F10 were all 0, and the standard deviation was also 0, which shows that HRLPSO can obtain the optimal value, 0, of the function every time it is optimized on these test functions, reflecting its better global optimization ability. Moreover, HRLPSO ranked first in the 12 test functions and the other 6 functions, as well as in the average ranking and final ranking. In the table, F represents the function name, D represents the dimension of the test function, Mean represents the average value of the objective function, and S.D. represents the standard deviation of the objective function value.

The average evolution curves under the 12 test functions are shown in Figure 5. In the 12 evolution curves, the final convergence accuracy of CLPSO on the test functions F1, F2, F3, F6, F7, F8, and F10 was better than that of PSO, but the convergence accuracy was worse than that of PSO when the number of iterations was 1000. Although HRLPSO did not converge when the number of iterations on test functions F1, F2, F3, F4, and F6 was 1000, it also achieved good convergence accuracy. It converged when the number of iterations on test functions F5, F7, F8, F9, F10, F11, and F12 was 1000. HRLPSO only converged after 1000 iterations of test functions F5, F6, F7, F11, and F12. Therefore, in this test function experiment, the number of iterations was set to 10000. Among the 12 evolution curves, HRLPSO had the highest convergence accuracy. Secondly, as shown in the figures, HRLPSO had the fastest convergence speed on unimodal functions F1, F2, F3, and F4 and multimodal functions F8, F9, and F10. By combining these results with the previous analysis regarding convergence accuracy, it can be concluded that HRLPSO not only has good convergence accuracy but also has good convergence speed.

Table 3. Performance indicators of different PSO algorithms.

Indicators	PSO	CIPSO	CLPSO	DLPSO	HRLPSO
Running times	30	30	30	30	30
Average computational time (s)	13.57	14.11	14.92	14.89	14.88
Average rank	4	3.89	2.78	2.44	1

presents a quantitative comparison of the performance indicators of the five algorithms, providing the average computational time and the average rank under 30 running times, achieved under the standard test functions F1~F12. From Table 3, it can be seen that the average rank of HRLPSO is the first and more advanced than the other algorithms. Meanwhile, although the average computational time of HRLPSO is shorter than that of CLPSO, DLPSO, it approximated that of the standard PSO and CIPSO. These indicators show that it performs best.

5.2. Test Results of the PSO Variants under CEC2013 Test Suite

The superiority-seeking performance of HDLPSO was verified in the previous experiments using 12 benchmark test functions. To make the optimization capability of HDLPSO more convincing, in this section, the experimental results of HDLPSO and the other four algorithms under the CEC2013 test suite are provided; see reference [30] for the specific test suite. The experimental parameters were set as the same values used in the previous experiments. In order to distinguish them from the previous 12 benchmark test functions, the 28 functions in the CEC2013 test suite were sequentially sorted by adding 12 to their names.

The optimization results of the five algorithms under the CEC2013 test suite are shown in

Table 4. Optimization results of HRLPSO and other algorithms under the CEC2013 test suite.

Functions	Dimensions	Indicators	PSO	CIPSO	CLPSO	DLPSO	HDLPSO
F13	30	M	1.21 × 104	−1.38 × 103	−1.01 × 103	−1.40 × 103	−1.40 × 103
		S	5.90 × 103	7.15	4.79 × 102	3.54 × 10−13	1.88 × 10−13
		R	6	2	3	1	1
F14	30	M	1.31 × 108	6.17 × 106	3.96 × 107	6.61 × 106	2.31 × 104
		S	8.72 × 107	2.37 × 106	2.76 × 107	3.74 × 106	2.25 × 104
		R	7	2	5	3	1
F15	30	M	6.53 × 1013	2.75 × 108	3.31 × 1010	2.36 × 109	3.22 × 108
		S	1.91 × 1014	1.44 × 108	1.83 × 1010	2.20 × 109	5.37 × 108
		R	7	1	4	3	2
F16	30	M	7.83 × 104	4.19 × 103	1.65 × 104	9.77 × 103	−6.81 × 102
		S	5.77 × 104	1.77 × 103	8.83 × 103	3.85 × 103	2.98 × 102
		R	7	2	4	3	1
F17	30	M	7.80 × 103	−9.80 × 102	−6.36 × 102	−1.00 × 103	−1.00 × 103
		S	5.16 × 103	9.81	5.56 × 102	1.37 × 10−9	1.14 × 10−13
		R	6	2	3	1	1
F18	30	M	1.02 × 103	−8.34 × 102	−8.30 × 102	−8.62 × 102	−8.81 × 102
		S	1.73 × 103	1.53 × 101	3.18 × 101	1.99 × 101	1.68 × 101
		R	7	3	4	2	1
F19	30	M	9.21 × 102	7.73 × 102	−6.88 × 102	−7.06 × 102	−7.27 × 102
		S	4.22 × 103	8.66	3.79 × 101	1.66 × 101	1.99 × 101
		R	7	1	4	3	2
F20	30	M	−6.79 × 102	6.79 × 102	−6.79 × 102	−6.79 × 102	−6.79 × 102
		S	5.67 × 10−2	5.05 × 10−2	6.81 × 10−2	4.32 × 10−2	6.91 × 10−2
		R	1	1	1	1	1
F21	30	M	−5.67 × 102	5.80 × 102	−5.62 × 102	−5.70 × 102	−5.78 × 102
		S	2.40	2.15	1.24	3.16	3.33
		R	5	1	6	3	2
F22	30	M	1.19 × 103	−4.76 × 102	−1.91 × 102	−4.88 × 102	−5.00 × 102
		S	9.47 × 102	1.32 × 101	1.79 × 102	1.74 × 101	4.31 × 10−2
		R	7	3	5	2	1
F23	30	M	1.74 × 101	−2.94 × 102	−3.54 × 102	−3.82 × 102	−3.64 × 102
		S	8.17 × 101	2.13 × 101	2.26 × 101	6.48	9.35
		R	7	4	3	1	2
F24	30	M	8.52 × 101	−1.91 × 102	−1.14 × 102	−1.95 × 102	−2.19 × 102
		S	9.44 × 101	2.08 × 101	2.32 × 101	3.29 × 101	1.85 × 101
		R	7	3	4	2	1
F25	30	M	1.71 × 102	7.48 × 101	-2.15 × 101	−4.37 × 101	−5.41 × 101
		S	7.00 × 101	2.08 × 101	1.62 × 101	3.04 × 101	3.26 × 101
		R	7	1	4	3	2
F26	30	M	6.50 × 103	4.33 × 103	2.26 × 103	1.47 × 102	1.16 × 103
		S	4.74 × 102	5.89 × 102	4.72 × 102	1.88 × 102	3.45 × 102
		R	7	6	3	1	2
F27	30	M	7.42 × 103	4.60 × 103	7.20 × 103	5.03 × 103	4.08 × 103
		S	3.63 × 102	5.32 × 102	3.28 × 102	6.75 × 102	5.63 × 102
		R	7	2	6	3	1
F28	30	M	2.02 × 102	2.02 × 102	2.02 × 102	2.02 × 102	2.01 × 102
		S	3.12 × 10−1	2.47 × 10−1	2.74 × 10−1	3.40 × 10−1	2.06 × 10−1
		R	2	2	2	2	1
F29	30	M	8.38 × 102	4.61 × 102	3.42 × 102	3.44 × 102	3.42 × 102
		S	1.41 × 102	3.09 × 101	2.51	4.72	7.69
		R	6	3	1	2	1
F30	30	M	8.66 × 102	5.72 × 102	5.87 × 102	5.52 × 102	4.87 × 102
		S	1.27 × 102	1.90 × 101	1.01 × 101	2.72 × 101	1.70 × 101
		R	7	3	4	2	1
F31	30	M	1.35 × 105	5.12 × 102	2.23 × 103	5.03 × 102	5.03 × 102
		S	2.64 × 105	2.26	2.50 × 103	1.04	1.25
		R	6	2	5	1	1
F32	30	M	6.13 × 102	6.11 × 102	6.12 × 102	6.14 × 102	6.12 × 102
		S	3.90 × 10−1	1.37	4.38 × 10−1	8.16 × 10−1	9.72 × 10−1
		R	3	1	2	4	2
F33	30	M	2.24 × 103	1.10 × 103	1.10 × 103	1.03 × 103	1.02 × 103
		S	5.14 × 102	5.13 × 101	1.65 × 102	1.53 × 102	5.91 × 101
		R	6	3	3	2	1
F34	30	M	7.96 × 103	5.09 × 103	2.89 × 103	1.38 × 103	2.02 × 103
		S	5.85 × 102	5.47 × 102	5.56 × 102	3.83 × 102	3.87 × 102
		R	7	4	3	1	2
F35	30	M	8.13 × 103	5.55 × 103	8.14 × 103	6.53 × 103	5.11 × 103
		S	4.65 × 102	7.54 × 102	2.75 × 102	6.16 × 102	8.07 × 102
		R	6	2	7	4	1
F36	30	M	1.30 × 103	1.26 × 103	1.28 × 103	1.28 × 103	1.27 × 103
		S	7.28	6.62	5.07	1.03 × 101	7.17
		R	5	1	3	3	2
F37	30	M	1.42 × 103	1.38 × 103	1.39 × 103	1.39 × 103	1.38 × 103
		S	1.45 × 101	1.07 × 101	9.56	7.53	8.22
		R	5	1	2	2	1
F38	30	M	1.56 × 103	1.47 × 103	1.50 × 103	1.40 × 103	1.40 × 103
		S	7.22 × 101	7.39 × 101	9.29 × 101	4.01 × 10−1	1.30 × 10−3
		R	5	2	3	1	1
F39	30	M	2.57 × 103	2.08 × 103	2.51 × 103	2.39 × 103	2.25 × 103
		S	1.16 × 102	7.64 × 101	9.04 × 101	8.60 × 101	9.18 × 101
		R	7	1	6	3	2
F40	30	M	4.69 × 103	1.87 × 103	3.18 × 103	2.08 × 103	1.76 × 103
		S	6.74 × 102	6.70 × 101	3.92 × 102	5.11 × 102	2.59 × 102
		R	7	2	4	3	1
Average R			5.96	2.18	3.71	2.21	1.36
Final R			7	2	4	3	1

, which show that the combined ranking of the five algorithms differed from the previous combined ranking under the 12 benchmark test functions. The ranking of HDLPSO, DLPSO, CLPSO, and PSO remained unchanged; they remained in first, third, fourth, and last place, respectively. Meanwhile, CIPSO ranked second overall; this reflects the fact that an algorithm cannot achieve the best results on every optimization problem. However, when considered together, the optimization results of the five algorithms under the CEC2013 test suite still showed that HDLPSO has excellent optimization capabilities.

6. Discussion

In order to improve the optimization ability of PSO, five improvement strategies were applied to the PSO algorithm. The reinforcement learning strategy in psychology was applied to the random generation of the initial population to leave better particles. The combination of cubic mapping and an adaptive strategy was applied to ω. This has the advantages of chaotic mapping and being adaptive at the same time. The adaptive strategy was used to adjust c1 and c2 to balance the individual learning ability and social learning ability of the algorithm. The dimension learning strategy was applied to improve the convergence speed and accuracy of the algorithm. Finally, Cauchy mutation and Gaussian mutation strategies were applied to the historical individual optimal solution and the historical global optimal solution, leaving better solutions to jump out of the local optimal solution. Using 12 benchmark functions, the algorithm and existing strategies were verified. The experimental results show that the proposed strategy has a good effect and prove the effectiveness and good optimization ability of the proposed strategy.

The future work is to further refine the HRLPSO algorithm as well as its parameters so that it can be applied to complex economic models.