A Self-Adaptive Vibration Reduction Method Based on Deep Deterministic Policy Gradient (DDPG) Reinforcement Learning Algorithm

Abstract

Although many adaptive techniques for active vibration reduction have been designed to achieve optimal performance in practical applications, few are related to reinforcement learning (RL). To explore the best performance of the active vibration reduction system (AVRS) without prior knowledge, a self-adaptive parameter regulation method based on the DDPG algorithm was examined in this study. The DDPG algorithm is unsuitable for a random environment and prone to reward-hacking. To solve this problem, a reward function optimization method based on the integral area of the decibel (dB) value between transfer functions was investigated. Simulation and graphical experimental results show that the optimized DDPG algorithm can automatically track and maintain optimal control performance of the AVRS.

1. Introduction

Due to the increasing requirement for accuracy and stability in research and manufacturing, the issue of micro-vibration, which widely exists in engineering applications, has become a critical factor that reduces the performance of high-precision machines. For example, as the state-of-art transmission electron microscope (TEM) has met the VC-G [1,2] or higher vibration standard (vibration velocity lower than 0.78 μm/s), the performance requirements of the AVRS are more stringent than before.

Active vibration control technology (AVCT) originated from the semi-active control methods represented by skyhook damping [3], for which it is difficult to obtain the absolute inertial reference frame (AIRF) that is required in practice. In a previous paper [4], a virtual skyhook damping isolator was proposed without the AIRF, which used a double cascade damping structure to suppress the resonance peak. In the engineering field, various high-precision instruments are equipped with passive vibration isolation systems, such as pneumatic or coil springs, which make them more sensitive to low-frequency vibrations. Because of the poor performance of passive and semi-active damping technology at low frequencies, the AVCT technique is urgently needed. The combination of AVCT and passive vibration isolators is one of the mainstream methods used today, in which passive isolators attenuate high-frequency vibrations while AVCT reduces the low-frequency vibrations. The AVRS overcomes the disadvantages of passive vibration reduction systems by integrating the active execution components.

Control algorithms [5,6,7,8] based on closed-loop PID have been proposed. Most of these methods are aimed at achieving the ideal dynamic capability of skyhook damping while balancing the robustness and stability of the system. Although appropriately increased feedback control (FBC) gain can enhance the vibration reduction effect, the adjustment range of parameters is greatly restricted by factors such as noise, decoupling accuracy, and ground and table disturbance [9]. Furthermore, the FBC response speed is greatly constrained by inherent algorithm delay and the hysteresis of the actuators.

The feed-forward control (FFC) strategy was first proposed to suppress the direct disturbances of AVRS, while the compensation control signal can be calculated from its regularity [10]. Under the premise of obtaining an accurate FF model, FFC can theoretically eliminate the influence of ground vibration on the platform. A previous paper [11] presented several prediction methods for FF signals and analyzed their tracking performance using filters. Irrespective of the FF signal prediction error changes with the increase in the frequency, proper parameters in FFC do not affect the closed-loop stability, which is merely determined by the loop transfer function [12].

To further improve the performance of AVRS, the FBC and FFC strategies were incorporated into a six-degree-of-freedom (DOF) isolation system that is based on absolute accelerator measurement [13], where FBC uses a genetic algorithm to suppress payload vibration, and FFC prevents ground vibration from transmitting to the platform.

Note that because of the hysteresis and the low response speed of the voice-coil motor, together with the high-order bending mode of the system, not only can the AVRS operating at fixed parameters be unstable, but it is also limited to suppressing the vibration below 100 Hz rather than the entire frequency range [14]. For the frequency above 100 Hz, vibration attenuation of AVRS mainly relies on passive isolation, while the AVCT focuses on the improvement of the low-frequency performance.

Furthermore, although the ideal model of AVRS is difficult to obtain, together with the rapid development of AVCT, various advanced control methods have been proposed for tracking the optimal parameters according to its specific applications [15,16,17]. However, few of these involve reinforcement learning (RL), which is one of the most popular branches in machine learning (ML) since Lee Sedol was defeated by AlphaGo in the board game Go. ML is excellent because it generates behavior directly from data of intelligent learning, rather than through complex programming [18]. RL is a trial-and-error learning algorithm with scarce guidance, whose goal is to train an agent to obtain the control objective or to enhance its performance through constantly interacting with the environment until achieving the best strategy, which is evaluated by the value of a reward function [19,20]. Although the Q-learning algorithm has been applied to realize automatic tuning of online parameters in the active structural control system [21], it is not suitable for continuous actions and its working frequency at 10 Hz is much lower than the ultra-precision vibration reduction range.

In this paper, based on the accurate mathematical derivation of the critical components in AVRS, a joint analysis model of ADAMS and MATLAB for AVRS is proposed. The mechanical structure of AVRS was presented elsewhere [13,14]. As the six DOF AVRS can be transformed into six independent single DOF sub-systems by the modal decomposition technique, a detailed single-channel block diagram of a vertical FB and FF control scheme can be produced by the transfer function of each component module, respectively. Then, the advanced DDPG algorithm is integrated into the FB and FF control scheme to determine the optimal control without considering the accuracy of the system model and the nonlinearity of parameters. DDPG, which is based on Actor-Critic [22] and Deep-Q-Network (DQN) algorithms [23] was proposed in [24]. It combines deep neural networks with a deterministic policy gradient (DPG) algorithm [25] and is one of the most popular algorithms for continuous actions. Simulation results show that the DDPG algorithm can achieve effective convergence and high stability in the optimal vibration reduction region. In practical applications, to overcome the invalidation of self-learning and the tendency toward reward-hacking of DDPG for random signals, in contrast to the tuning methods proposed in [26,27,28], this study introduced an optimal calculation method of the reward function. Experiment results validated that the agent trained by this method has superior decision-making performance and excellent vibration damping effects.

2. Principle of Active Vibration Isolation

2.1. The Principle of Active Vibration Control

The single DOF mass-spring damping system with active control used in this research is shown in Figure 1.

The vibration signals are measured by absolute velocity sensors and the control forces are generated by voice-coil motors. When ignoring the gravity and the static displacement caused by it, the linear ordinary differential equations in the equilibrium position of the system can be rewritten as below:

$$m\frac{\mathrm{d}{v}_{\mathrm{p}}}{\mathrm{d}t}+c\left(v_{\mathrm{p}}-v_{\mathrm{b}}\right)+k\left({\displaystyle \int v_{\mathrm{p}}}\mathrm{d}t-{\displaystyle \int v_{\mathrm{b}}}\mathrm{d}t\right)=f(v_{\mathrm{p}},v_{\mathrm{b}})+f_{\mathrm{noise}}$$

(1)

where $v_{\mathrm{p}}$ and $v_{\mathrm{b}}$ are the measured absolute velocity of the payload and ground, respectively; $m$, $c$, and $k$ are the system payload, equivalent damping, and stiffness, respectively; $f_{\mathrm{noise}}$ is the plate-top disturbance signal; $f\left(v_{\mathrm{p}},v_{\mathrm{b}}\right)$ is the motor driving force function with $v_{\mathrm{p}}$ and $v_{\mathrm{b}}$ as the dependent variables. Since the vibration amplitude of AVRS applied to ultra-precision equipment is small, the inverse electromotive force generated by the velocity difference between the coils and the magnet in the voice-coil motor can be ignored, so that the thrust coefficient $K_{\mathrm{T}}$ of the voice-coil motor is approximately constant. Then, the voice-coil motor output force $f\left(v_{\mathrm{p}},v_{\mathrm{b}}\right)$ can be considered to have an approximately linear relationship with $v_{\mathrm{p}}$ and $v_{\mathrm{b}}$.

2.2. FB and FF Control Theory

Enlightened by the idea of independent channel control design that was based on the different disturbance transmission paths [29], and referring to the principle of force superposition of the FB and FF hybrid control system, the control force, generated by the voice-coil motor, can be decomposed into two parts: one provided by the FFC and the other by the FBC. It is worth noting that the driving forces of the FBC and FFC are not independent of each other in practical applications. It was found that the FFC will effectively prevent the ground disturbance that will transmit to the platform [11], and the corresponding FBC output force becomes smaller because the vibration amplitude of the payload decreases.

Here, $s$ represents the Laplace operator. Assuming that the initial speed of the platform is zero, and taking $f_{\mathrm{noise}}$ and $v_{\mathrm{b}}\left(s\right)$ as external disturbance sources, the value of FF force $f_{\mathrm{ffc}}\left(s\right)$ is determined by $v_{\mathrm{b}}\left(s\right)$ and has nothing to do with $v_{\mathrm{p}}\left(s\right)$. Without considering the nonlinear factors of sensors, signal amplifier circuits, and actuators, the relationship between $f_{\mathrm{ffc}}\left(s\right)$ and the ground detection signal $v_{\mathrm{b}}\left(s\right)$ can be expressed as follows:

$$f_{\mathrm{ffc}}\left(s\right)=H_{\mathrm{ffc}}\left(s\right)v_{\mathrm{b}}\left(s\right)$$

(2)

where $H_{\mathrm{ffc}}\left(s\right)$ denotes the velocity–force transfer function.

Now, the dynamic characteristics of the AVRS in terms of two aspects during the feedback control are investigated in the following paragraphs.

(1) When the AVRS is open-looped, according to the principle of speed superposition, the platform speed signal $v_{\mathrm{p}}\left(s\right)$ can be decomposed into two parts, $v_1\left(s\right)$ and $v_2\left(s\right)$. Here, $v_1\left(s\right)$ is the component of platform velocity transmitted from ground disturbance $v_{\mathrm{b}}\left(s\right)$ under the control of $f_{\mathrm{ffc}}\left(s\right)$, while $v_2\left(s\right)$ is caused by $f_{\mathrm{noise}}$. Then, the expression of the system’s kinematic transfer functions described in Figure 1 is given by the Equation (3):

$$\{\begin{array}{l}ms{v}_1\left(s\right)+c\left[v_1\left(s\right)-v\left(s\right)\right]+\frac{k}{s}\left[v_1\left(s\right)-v\left(s\right)\right]=-f_{\mathrm{ffc}}\left(s\right)\\ ms{v}_2\left(s\right)+c{v}_2\left(s\right)+\frac{k}{s}{v}_2\left(s\right)=f_{\mathrm{noise}}\left(s\right)\end{array}$$

(3)

The equations to calculate $v_1\left(s\right)$ and $v_2\left(s\right)$ can be presented as follows:

$$\{\begin{array}{l}{v}_1\left(s\right)=\frac{\left(cs+k\right)-s{H}_{\mathrm{ffc}}\left(s\right)}{m{s}^2+cs+k}{v}_{\mathrm{b}}\left(s\right)\\ v_2\left(s\right)=\frac{f_{\mathrm{noise}}\left(s\right)s}{m{s}^2+cs+k}\end{array}$$

(4)

Therefore, when the system is open-looped, the platform speed can be obtained by:

$$v_{\mathrm{p}}\left(s\right)=v_1\left(s\right)+v_2\left(s\right)$$

(5)

By substituting Equation (4) into Equation (5), finally, $v_{\mathrm{p}}\left(s\right)$ can be expressed by Equation (6) as below:

$$v_{\mathrm{p}}\left(s\right)=\frac{s}{m{s}^2+cs+k}\left[\left(\frac{cs+k}{s}-H_{\mathrm{ffc}}\left(s\right)\right)v_{\mathrm{b}}\left(s\right)+f_{\mathrm{noise}}\left(s\right)\right]$$

(6)

Considering the combined effects of $v_{\mathrm{b}}\left(s\right)$ and $f_{\mathrm{noise}}\left(s\right)$, the FB force $f_{\mathrm{fbc}}\left(s\right)$ can be calculated by $H_{\mathrm{fbc}}\left(s\right)$ multiplied by $v_{\mathrm{p}}\left(s\right)$, which is expressed in Equation (7) below:

$$f_{\mathrm{fbc}}\left(s\right)=H_{\mathrm{fbc}}\left(s\right)v_{\mathrm{p}}\left(s\right)$$

(7)

where $H_{\mathrm{fbc}}\left(s\right)$ stands for the forward channel transfer function between the platform velocity $v_{\mathrm{p}}\left(s\right)$ and feedback force $f_{\mathrm{fbc}}\left(s\right)$. Then, substituting Equation (6) into Equation (7), yields:

$$f_{\mathrm{fbc}}\left(s\right)=H_{\mathrm{fbc}}\left(s\right)\frac{s}{m{s}^2+cs+k}\left\{\left[\frac{cs+k}{s}-H_{\mathrm{ffc}}\left(s\right)\right]v_{\mathrm{b}}\left(s\right)+f_{\mathrm{noise}}\left(s\right)\right\}$$

(8)

From Equation (8), it can be seen that $f_{\mathrm{fbc}}\left(s\right)$ contains the term $H_{\mathrm{ffc}}\left(s\right)$, which indicates that the influence of ground disturbance has not been eliminated by $f_{\mathrm{ffc}}\left(s\right)$. Therefore, the platform residual disturbance from ground disturbance, together with $f_{\mathrm{noise}}\left(s\right)$, will be suppressed by the force component $f_{\mathrm{fbc}}\left(s\right)$. Thus, before the closed-loop control is applied, the motor output force $f_{\mathrm{M}}\left(s\right)$ is calculated as follows:

$$f_{\mathrm{M}}\left(s\right)=f_{\mathrm{ffc}}\left(s\right)+f_{\mathrm{fbc}}(s)$$

(9)

According to the above formula, the relationship between the motor driving force $f_{\mathrm{M}}\left(s\right)$ and ground vibration $v_{\mathrm{b}}\left(s\right)$, together with surface disturbance $f_{\mathrm{noise}}\left(s\right)$, can be obtained as below:

$$\begin{array}{cc}{f}_{\mathrm{M}}\left(s\right)& =H_{\mathrm{fbc}}\left(s\right)\frac{s}{m{s}^2+cs+k}\left\{\frac{cs+k}{s}+\frac{m{s}^2+\left[c-H_{\mathrm{fbc}}\left(s\right)\right]s+k}{H_{\mathrm{fbc}}\left(s\right)s}{H}_{\mathrm{ffc}}\left(s\right)\right\}{v}_{\mathrm{b}}\left(s\right)\hfill \\ & +H_{\mathrm{fbc}}\left(s\right)\frac{s}{m{s}^2+cs+k}{f}_{\mathrm{noise}}\left(s\right)\hfill \end{array}$$

(10)

(2) In the closed-loop system, the motor force acts on the system at the value $f_{\mathrm{M}}\left(s\right)$ shown in Equation (10), and the table vibration is rapidly attenuated due to the effect of $f_{\mathrm{M}}\left(s\right)$. As the output of the controller remains the same before being updated, the dynamic process of the system from the perspective of computer control can be analyzed. When ignoring the response delay of the system, assuming that at some point after the control loop is closed, the platform velocity under the combined effect of the motor driving force ${f_{\mathrm{M}}}^{\prime }\left(s\right)$, $v_{\mathrm{b}}\left(s\right)$, and $f_{\mathrm{noise}}\left(s\right)$ is ${v_p}^{\prime }\left(s\right)$. This means that dynamic Equation (1) still holds. Then, the following equations can be derived:

$$\{\begin{array}{l}ms{v_{\mathrm{p}}}^{\prime }\left(s\right)+c\left[{v_{\mathrm{p}}}^{\prime }\left(s\right)-v_{\mathrm{b}}\left(s\right)\right]+\frac{k}{s}\left[{v_{\mathrm{p}}}^{\prime }\left(s\right)-v_{\mathrm{b}}\left(s\right)\right]=f_{\mathrm{noise}}\left(s\right)-{f_{\mathrm{M}}}^{\prime }\left(s\right)\\ {f_{\mathrm{M}}}^{\prime }\left(s\right)=H_{\mathrm{ffc}}\left(s\right)v_{\mathrm{b}}\left(s\right)+H_{\mathrm{fbc}}\left(s\right){v_{\mathrm{p}}}^{\prime }\left(s\right)\end{array}$$

(11)

Simplifying the above equations, ${v_p}^{\prime }\left(s\right)$ can be obtained in Equation (12):

$${v_p}^{\prime }\left(s\right)=\frac{f_{\mathrm{noise}}\left(s\right)+\left[\left(c+k/s\right)-H_{\mathrm{ffc}}\left(s\right)\right]v_{\mathrm{b}}\left(s\right)}{ms+c+k/s+H_{\mathrm{fbc}}\left(s\right)}$$

(12)

Theoretically, if the system is ideal, the model of FF can be identified as:

$$H_{\mathrm{ffc}}\left(s\right)=\frac{cs+k}{s}$$

(13)

From the derived formula, it is observed that FFC can eliminate the impact of ground vibration on the system. Therefore, rather than deteriorating the system’s stability, proper FFC will improve the robust performance of the system. To better reflect the influence of FF and FB parameters on the damping performance of AVRS, the vertical vibration transmissibility curve, which is used to analyze the relationship between the ground disturbance $v_{\mathrm{b}}\left(s\right)$ and the platform vibration ${v_{\mathrm{p}}}^{\prime }\left(s\right)$ while ignoring $f_{\mathrm{noise}}\left(s\right)$, can be rearranged from the Equation (12) as:

$$\frac{{v_{\mathrm{p}}}^{\prime }\left(s\right)}{v_{\mathrm{b}}\left(s\right)}=\frac{\left(c+k/s\right)-H_{\mathrm{ffc}}\left(s\right)}{ms+c+k/s+H_{\mathrm{fbc}}\left(s\right)}$$

(14)

Compared with the passive curve shown by the red line in Figure 2, the pure FF proportional (P) control effect in the dotted red line is not ideal because it amplifies the amplitude at the resonant frequency point while suppressing the vibration at high frequencies. The optimal coefficient value of P is a trade-off between the high- and low-frequency damping performance. Therefore, merely P in FFC is not sufficient, especially when considering the high-performance requirements of ultra-precision equipment.

It can be seen from the dotted gray line that the integral (I) control mode in FFC has a better attenuation effect in the low-frequency domain. Therefore, as shown in the dotted green line, it is necessary to adopt the proportional plus integral (PI) control mode in FFC, rather than P or I mode. It is worth noting that the simulation results show that the FFC has a very limited ability to suppress vibration at the structural resonance frequency. As indicated in the dotted blue curve in the figure, the closed-loop P control was added based on feed-forward PI control, which can effectively suppress the natural frequency of the platform. It can be seen from the pink line and dotted black line in Figure 2 that, with the increase in P, the vibration attenuation effect is better. Not only can the adoption of the closed-loop PI mode regulate the equivalent damping, but it changes the equivalent stiffness of the system. Although the performance of closed-loop PI control is superior to P in the frequency range below 1 Hz, due to the constraints of the ultra-low frequency detection accuracy of sensors, the closed-loop system usually adopts the P control mode.

In the absolute velocity feedback control system, although the performance of FBC will be improved as the feedback control gain increases, the system will be unstable at high gains if the influence of the model, the bandwidth of sensors and actuators, the sampling rate, etc., are considered [30,31]. In addition, under the control of fixed parameters, AVRS may be unstable due to load changes, environmental interference, human operation, and other factors. To ensure stability and improve the performance of AVRS, this study aimed to introduce the DDPG reinforcement learning algorithm to realize the automatic optimization of control parameters through continuous interaction with the working environment.

3. MATLAB and ADAMS Co-Simulation

3.1. The Single DOF Model of AVRS in the Vertical Direction

Due to the complex coupling characteristics of the parameters in the system having six degrees of freedom, the AVRS should be transformed into six independent subsystems. To conduct the co-simulation, ADAMS and MATLAB were employed to establish a single DOF model of AVRS in the vertical direction to study the characteristics of FFC and FBC. In this study, ADAMS software was used to simulate the main structure of the system, which is supported by four parallel steel springs, and MATLAB was used to build the control system. The vertical driving force was established at the center of each spring to simulate the vertical linear motor in the actual system. By applying white noise excitation to the base, the passive velocity transmissibility curve, which is critical for model validation, was acquired and is shown in Figure 3.

The black dotted line represents the passive transfer function curve that was obtained from the sweep frequency test of the ADAMS model. The curve is consistent with the mathematical transfer function model shown by the green line in the same figure. The red line is the experimental transmissibility curve that was vertically acquired by external VSE-15D sensors and the LMS-SCADAS test system. The results demonstrate that the simulation model can still well fit the realistic system below 40 Hz even considering the influence of the system’s high-frequency noise and high-order modes.

3.2. DDPG Reinforcement Learning Algorithm

The DDPG algorithm is an upgraded version of the Actor-Critic (AC) algorithm shown in Figure 4. It integrates the advantages of DQN and overcomes the shortcomings of the Actor-Critic algorithm. It uses a replay memory buffer and soft-update target networks to realize stable and robust control in large-scale states or continuous action space. In addition, it adopts the mini-batch method to eliminate the correlation between samples. Through batch normalization to gain hyper-parameters, the agent is enabled to generalize across the environments with different scales of state values. The following paragraphs briefly describe the DDPG algorithm process according to the control flow presented in Figure 4. Details were previously presented elsewhere [24,32]. It is worth noting that the DDPG algorithm adopts the AC structure which includes a policy-based neural network (Policy-net) system and a value-based neural network (Q-net) system. Each consists of an online network and a target network. The two target networks are updated using the same software approach, while the two online networks are different: the online Q-net updates the parameters by minimizing the mean squared error (MSE) as the loss function (LF) to obtain the maximum value of Q, whereas the online Policy-net updates the network parameters by training the mini-batch data to achieve an unbiased estimate of the DPG according to the Monte Carlo method.

Here, the LF can be expressed as follows:

$$LF=\frac{1}{\mathrm{N}}{\displaystyle {\sum }_i{\left[\underset{TargetQ-net: \mathit{6}\to \mathit{7}}{\underbrace{r_i+\gamma Q^{\prime }\left(S_{i+1},\underset{Targetpolicy-net: \mathit{5}\to \mathit{6}}{\underbrace{{\mu }^{\prime }\left(S_{i+1}|{\theta }^{{\mu }^{\prime }}\right)}}|{\theta }^{Q^{\prime }}\right)}}-\underset{OnlineQ-net: \mathit{6}\to \mathit{7}}{\underbrace{Q\left(S_i,a_i|{\theta }^Q\right)}}\right]}^2}$$

(15)

and the DPG can be written as shown in Equation (16):

$${\nabla }_{{\theta }^{\mu }}\mu |S_i\approx \frac{1}{N}{\displaystyle \sum _i\underset{DPG: \mathit{11}\to \mathit{12}}{\underbrace{{\nabla }_a\underset{OnlineQ-net: \mathit{9}\to \mathit{10}\to \mathit{11}}{\underbrace{{Q\left(S,a|{\theta }^Q\right)|}_{S=S_i,a=\mu (S_i)}}}{\nabla }_{{\theta }^{\mu }}\underset{OnlinePolicy-net: \mathit{9}\to \mathit{11}}{\underbrace{{\mu \left(S|{\theta }^{\mu }\right)|}_{S_i}}}}}}$$

(16)

Thus, according to the subscript in Equations (15) and (16) and the structure in Figure 4, the calculation process of the DDPG algorithm can be clearly understood. In this paper, the adaptive moment estimation (Adam) optimizer used in the online neural networks of DDPG enables faster convergence to the global optimum.

3.3. Simulation of DDPG Algorithm in Active Vibration Control

To build a simulation model that is consistent with the realistic AVRS, the motor drive model $M\left(s\right)$ and the low-frequency extensive model of the sensor $S\left(s\right)$ should be considered. Suppose that the control blocks of FF and FB are $FF\left(s\right)$ and $FB\left(s\right)$; then, the classic structure of AVRS is shown in the black-dotted box in Figure 5.

Based on the aforementioned FB and FF co-simulation control model, the DDPG algorithm module was constructed in MATLAB. Through the plant export function, the M file exported from the model of ADAMS was imported into MATLAB to obtain the environment required for the simulation and to realize the information exchange between ADAMS and MATLAB.

Through continuous iterative analysis of the vibration reduction performance of AVRS, the critical control parameters of $FB\left(s\right)$ and $FF\left(s\right)$ were updated to make the neural network learning parameters converge to the global optimum.

3.3.1. The Structure of the Neural Network

The neural network structure of online Policy-net and online Q-net in DDPG is shown in Figure 6.

Online Policy-net has one hidden layer with fifty nodes and receives the current state signal $S$ and outputs the action signal $a$ after the Gaussian distribution and amplitude restriction. The activation function of hidden layers in the online Policy-net is Relu, and that of the output layer is also Relu. The value of the online Q-net has the same numbers of hidden layers and nodes as in the online Policy-net and represents the action reward, which is used to evaluate the effectiveness of the selected action. The activation function of the hidden layer in online Q-net is Tanh, and that of the output layer is linear-activated.

3.3.2. Reward Function

The setting of the reward function is crucial; a good design can not only speed up the training process of the reinforcement learning algorithm but also avoid the situation of reward-hacking and convergence to a local optimum. The control objective for active vibration reduction is that the absolute speed of the payload is infinitely close to zero. Then, the reward function can be set as a linear function of $v_{\mathrm{p}}$ in simulation, and its expression is shown in Equation (17):

$$R=-\left|v_{\mathrm{p}}\right|-0.1v_{\mathrm{p}}{}^2$$

(17)

3.4. The Results of Simulation

In the simulation, white noise signals with amplitudes of 1 and 200 were used to simulate ground excitation and table disturbance, respectively. More detailed parameters are shown in

Table 1. Specifications of simulation.

MATLAB
Solver	ode4	Type	Fixed-step
Episode time	2s	Fixed-step size	0.001s
ADAMS Model
ADAMS Solver type	Fortran	Simulation mode	Continuous
Animation mode	batch	Communication interval	0.001s
DDPG
TargetSmoothFactor	1 × 10−3	Agent sample time	0.001s
MaxEpisodes	10,000	MaxStepsPerEpisode	20
MiniBatchSize	64	DiscountFactor	0.998

.

In this study, the DDPG reinforcement learning algorithm simulation was carried out based on the joint control model, which mainly includes the following two parts: (1) FBC parameter that is optimized through DDPG under the condition of proper FFC parameter; (2) FFC and FBC parameters optimize the system simultaneously.

From Figure 7, EpisodeQ0 stands for the prospective discount reward, whose stability reflects whether the critic network was designed properly. The reinforcement learning reward curve of the FBC proportional parameter under the specified feed-forward parameter is shown in Figure 7a, and the self-learning episode reward curve of multiple parameters that contains both the feed-forward gain and the feedback gain is shown in Figure 7b. The result shows that when multiple parameters are learned at the same time, the system can still converge to the optimal value range in a long step size.

4. Experiments

The DDPG reinforcement learning algorithm, which is a type of model-free RL method, achieves optimal control through continuous interaction with the environment, receives action from the agent and returns the corresponding reward value of the current state. The agent only needs to collect and analyze the vibration signals from the ground and platform to realize self-learning control. The purpose of AVRS is to minimize the vibration of the platform by regulating the real-time dynamic parameters and changing the output control force value corresponding to the detection signals. However, in practical applications, considering the nonlinear characteristics of the inertial system and its multivariable coupling, it is difficult to accurately evaluate the system performance when the updated frequency of parameters is too high.

To improve the control robustness of the system and realize the engineering application of the DDPG control method in the field of vibration reduction, this study aimed to transform real-time dynamic parameter regulation into optimal parameter adjustment. Optimizing the reward function overcomes the shortcomings of the DDPG algorithm, which is highly dependent on the system operating speed and hardware performance, and makes the intelligent integration of the system more economical.

4.1. Reward Function Optimization

Not only the DDPG algorithm is unsuitable in a random environment, but it facilitates reward-hacking when the reward function is improperly set in the process of reinforcement learning. Since the transmissibility curve reflects the modal information about AVRS, and its corresponding technical indicators are not affected by the form of environmental excitation signal under the same parameters, it is appropriate for the amplitude curve of transmissibility to be set as the reference of the reward function. By integrating the difference decibel value of the amplitude between the passive transmissibility curve and the active control curve within the specified frequency range, the area size of the enveloped curve directly reflects the attenuation performance of the AVRS.

Then, the reward function is defined as:

$$R=\{\begin{array}{cc}0& f_i< f_{\mathrm{H}},f_i\ge f_{\mathrm{H}}\\ {\displaystyle \sum _{i=0}^n\left\{\left[F_{\mathrm{o}}\left(f_i\right)-F_{\mathrm{c}}\left(f_i\right)\right]\ast \left[lg\left(f_{i+1}\right)-lg\left(f_i\right)\right]\right\}}& f_{\mathrm{L}}< f_i< f_{\mathrm{H}}\end{array}$$

(18)

In Equation (18), $F_{\mathrm{o}}\left(f_i\right)$ and $F_{\mathrm{c}}\left(f_i\right)$ represent the amplitude of passive and active control transmissibility at $f_i$, respectively. According to the requirements of different technical indicators, the corresponding boundary frequency range $\left[f_{\mathrm{L}},f_{\mathrm{H}}\right]$ and the calculation method can be modified, and the parameters optimized through the DDPG self-learning algorithm, to achieve the control performance that meets the application requirements.

4.2. Experimental Setup

The AVRS experimental setup consists of four independent vibration isolation units, controller, payloads, PC, etc. Each isolator is integrated with a steel spring, geophones, and voice-coil motors. The schematic diagram of the experiment is shown in Figure 8.

In this experiment, excitation signals generated by the LMS signal system were amplified and acted on the AVRS substrate by the MB shaker. All internal signals were acquired by the AD port in the DSP controller and, after frequency expansion, signal decoupling, and algorithm processing, they were output from the DA port as motor drive signals.

In addition, the PC was used to establish an intelligent agent, which communicates with the DSP in the controller through the serial port and the CAN port. The CAN port realizes vibration signal transmission, and the serial port is used to execute the parameter’s updated commands. The online network parameters are updated at the end of each 100-step episode. Before starting self-learning control, 2000 random samples should be stored in replay memory. When the sample size reaches the set value, the agent randomly selects 32 mini-batch samples for DDPG reinforcement learning. Next, two external high-resolution VSE-15D servo velocity seismometers are used to verify the vibration reduction effect of the AVRS.

Table 2. Specifications of components in AVRS.

Components	Symbol	Parameter	Value
Payload	$m$	Mass	1750 kg
Helical spring	$k$	Stiffness	300,000 N/m
	$c$	Damping	490 Ns/m
Voice-coil motor	KT	Force constant	80 N/A
	RM	Resistant	34.5 Ω
	LM	Inductance	28 mh
		Moving distance	28 mm
		Frequency range	≤200 Hz
Geophone	$f_g$	Natural frequency	4.5 Hz
	$G_g$	sensitivity	100.4 V/m/s
	$R_d$	Internal resistant	2450 Ω

shows the parameters of the AVRS.

4.3. Experimental Results

The content of the experiment mainly comprises the following three parts: (1) the FFC parameter, which is optimized through DDPG under the condition of a proper FBC parameter; (2) the FBC parameter is optimized in a reasonable FFC parameter; (3) FFC and FBC parameters optimizing simultaneously.

Figure 9 shows the experimental results of FFC parameter self-learning by the DDPG algorithm when the FBC proportional gain is fixed at 1.8. Figure 9a–e are the learning curves of the 0/20/40/80/100th episode, where the horizontal axis is the number of steps, the red line is the value of the FFC parameter, and the blue line is the corresponding value of the action reward. Figure 9f reflects the convergence of the FFC parameter in the whole self-learning process.

Figure 10 shows that there is an approximate normal envelope distribution relationship between the FFC parameter and the state reward value. When the parameter is located between [−15, −5], the active vibration reduction performance becomes better with the increase in the FFC parameter. When it is bigger than −4, the control performance gradually deteriorates. When the parameter is at the limit value of −15 or 5, the system state value is negative, indicating that the system is in a state of vibration amplification or instability. The parameter range of the maximum state reward value is [−5, −4], as marked in the red circle, which is consistent with the region corresponding to the parameter convergence value in Figure 9f, indicating that the DDPG algorithm can track the optimal value range of single-parameter FFC during self-learning.

As shown in Figure 11, the FFC proportional control parameter is fixed to −4.5, and the DDPG optimization learning is carried out when the initial value of the FBC proportional parameter is 0. Figure 11a–e are also the learning curves of the 0/20/40/80/100th episode. Figure 11f reflects the convergence range of the FBC parameter at [2, 3].

The experimental learning curve in Figure 12 can be divided into three regions: the stable region, the transition region, and the unstable region. It is observed that when the FBC parameter is less than 2.5, the system is in the stable region, and with the increase in the FBC parameter, the control performance of the system improves gradually. In the transition region of [2.5, 6], it is found that there is a similar hysteresis characteristic between the system state reward and the FBC proportional parameter. Therefore, when the FBC parameter increases monotonously from 2.5 to 4.5, the system performance deteriorates. When it is greater than 4.5, the vibration reduction performance of the system starts to decline sharply, and the vibration amplitude of the platform is constant and amplified compared with the initial value. When the FBC parameter increases to above 6, the system is in an unsteady state. Conversely, when the parameter decreases monotonously from 6 to 4.5, the system is still unstable; when it is less than 3, the system gradually returns to the stable state.

In Figure 13, the results of the multi-parameter self-learning experiment are shown when the initial proportional control parameters P of the FBC and FFC are both 0. From Figure 13a–f, the optimal parameter of FBC is in the range of [2.0, 3.0], and the FFC parameter is located in [−5.0, −4.0]. The results show that the parameters learned simultaneously have good consistency with the values learned separately.

Figure 14 is the 3D projected scatter diagram of state reward and the multiple parameters. The red circles in the figure indicate the area where the optimal values are located, and the identification results are consistent with the final convergence values of the learning curves. Due to the high-order, nonlinearity, and strong coupling characteristics of the actual system, its dynamic characteristics are very complex, and it is difficult to obtain accurate system control parameters. Nevertheless, the above experiments show that a small optimal value interval can be determined.

Based on the DDPG self-learning results above, which take the integral area of the difference between the passive and active vibration transmissibility curves as the reward function, the vertical FBC parameter of the system is selected as 2.5 and the FFC parameter as −4.0.

Figure 15 shows the comparison results of vibration transmissibility curves about internal geophones and external VSE-16D sensors under passive, FFC, FBC, and hybrid control. The curves in the figure show that the decoupling calculation value of the internal distributed sensors, which can correctly reflect the actual vibration of the system, has good consistency with the direct test value of the external sensors. The experimental results show that the application of the DDPG algorithm in active vibration control is feasible and effective.

Through the DDPG control method, the influence of the control parameters on the system performance within the set range can be analyzed. When the system structure and payload change, the agent can repeat the learning process until the new optimal control parameters are obtained.

In addition, the final learned parameters can be easily solidified into the static flash to achieve high-performance offline operation. The theoretical results of the method proposed in this paper are in good agreement with the experimental measurements and have strong control robustness in the active vibration reduction field. This method has good application prospects in mature commercial products.

5. Conclusions

In this study, the dynamic control mechanism of FB and FF was investigated, and a detailed block diagram of the FB and FF control system was proposed according to the classification of vibration transmission paths. In addition, based on the united FF and FB control case, the DDPG algorithm was introduced to perform real-time self-learning on the system parameters. Since DDPG is not suitable for random signal learning, this paper proposed a reward function optimization method that is still applicable to the multi-parameter self-learning process. Simulations and experiments show that the DDPG algorithm can continuously optimize system parameters through interaction with the isolator units to achieve effective suppression of platform vibration without the need for artificial prior knowledge. Therefore, the method presented in this paper has great potential value in future practical applications because it avoids human interference. The reward function optimization is crucial in speeding up the DDPG learning process. The setting method of the reward function proposed in this paper still has the disadvantages of random errors at low frequencies and calculation errors at high frequencies. Therefore, reasonable value selection and optimal calculation of the frequency interval will be important directions in subsequent research.