Trajectory Tracking Control Based on Deep Reinforcement Learning for a Robotic Manipulator with an Input Deadzone

Abstract

This paper proposes a deep reinforcement learning (DRL) method that combines random network distillation (RND) and long short-term memory (LSTM) to address the tracking control problem, while leveraging the inherent symmetry in robotic arm movements to eliminate the need for learning or knowing the system’s dynamic model. In general, the complexity and strong coupling of robotic manipulators make trajectory tracking extremely challenging. Firstly, the prediction network and fixed network are jointly trained using the RND method. The difference in output values between the two networks acts as an internal reward for the robotic manipulator environment. This internal reward mechanism encourages the robotic arm agent to actively explore unpredictable and unknown environmental states, thereby consequently boosting the performance and efficiency of the tracking control for the robotic manipulator. Then, the Soft Actor-Critic (SAC) algorithm, the LSTM network, and the attention mechanism are integrated to resolve the instability problem during training and acquire a stable policy. The LSTM model effectively captures the symmetry and temporal changes in joint angles, while the attention mechanism dynamically prioritizes important features, thereby reducing the instability of the robotic manipulator during tracking tasks and enhancing feature extraction efficiency. The simulation outcomes demonstrate that the proposed method effectively performs the robot tracking task, confirming the efficacy and efficiency of the DRL algorithm.

1. Introduction

The advent of artificial intelligence (AI) has significantly impacted various fields, including the control of robotic manipulators [1]. The control of manipulators is a complex task that requires precision and adaptability to varying environments and tasks. Traditional control methods have been widely used, offering robust solutions for specific, well-defined problems. Several approaches are proposed to attain satisfactory tracking control results, including sliding mode control, PID control, model predictive control, and adaptive tracking control [2,3,4,5,6,7]. However, these methods often lack the flexibility and learning capabilities needed for more dynamic and unpredictable environments. Fortunately, neural networks, with the ability to learn from data and generalize to new situations, are employed to enhance the performance of robotic systems. In [8], an adaptive tracking control policy based on neural networks is proposed for robot manipulators. In [9], a new control method based on neural networks is proposed to control the robotic manipulator, utilizing neural networks to approximate its dynamic model. In [10], the development and experimental verification of a dual-loop nonlinear controller for a quadrotor, based on an adaptive neural network, is presented.

However, neural networks often require vast amounts of labeled training data and can struggle with generalizing to entirely new scenarios without retraining. They are prone to overfitting and may not perform well in highly dynamic or unpredictable environments. Reinforcement learning (RL) has attracted attention for its applications in environments where systems can learn optimal control policies through trial and error. In [11], an algorithm utilizing Q-learning (QL) and deep Q networks (DQN) is introduced for training a rotary inverted pendulum system. The SARSA (State-Action-Reward-State-Action) algorithm is applied to control the positioning of the end effector of a three-degree-of-freedom arm, targeting both fixed and random points in [12]. While QL, DQN, and SARSA are useful for addressing specific RL tasks, their primary application is in discrete spaces. Robotic arms need to operate with a high degree of precision, requiring smooth and continuous control signals that discrete algorithms cannot efficiently provide. Continuous RL algorithms have further advanced the field by allowing control in continuous action spaces, being more natural for a robotic manipulator. In [13], a deep reinforcement learning (DRL) control method is proposed for position and attitude tracking of bionic underwater vehicles (BUV), where the Soft Actor-Critic (SAC) algorithm is used to train the controller through interaction with a simulated BUV.

However, in highly coupled and complex environments such as robotic arms, these algorithms often exhibit significant instability during training. This is where the integration of long short-term memory (LSTM) neural networks and an attention mechanism with SAC can effectively mitigate these instabilities. LSTM networks are designed to handle sequential data and maintain long-term dependencies, making them ideal for processing the time-series data that are prevalent in robotic control tasks [14]. In [15], the SAC algorithm and the random network distillation (RND) method are combined to control a robotic arm to capture the target. The internal rewards generated by RND often motivate the agent to explore more extensively to speed up the convergence of the algorithm.

This paper aims to draw on the advantages of the LSTM network, attention mechanism, and RND to design a DRL trajectory tracking controller for a robotic manipulator with a deadzone. The primary contributions are summarized in the following points.

(1) By generating intrinsic rewards based on the prediction error of a neural network, RND motivates the agent to explore more extensively [15]. This extensive exploration accelerates the convergence of the algorithm by encouraging the agent to visit a wider variety of states, which is crucial in complex robotic tasks.

(2) In highly coupled and dynamic environments like robotic manipulators, maintaining stable learning is challenging. The LSTM component processes time-series data efficiently, capturing the dependencies between consecutive actions and states [14], and the attention mechanism can dynamically focus on critical features. This capability significantly mitigates the instability issues during training, leading to more reliable and consistent policy performance.

(3) SAC provides a robust framework for learning policies in continuous action spaces, ensuring smooth and precise movements. The LSTM component and the attention mechanism enhance the ability of the algorithm to handle the temporal dependencies and stabilize training. RND introduces a mechanism for intrinsic motivation, driving the agent to explore and learn more efficiently. Together, these components create a powerful and stable control strategy for robotic arms, capable of executing complex tasks with high precision and reliability.

2. System Description

Dynamics Model

By deriving equations from the Lagrangian function, we can obtain the dynamic equations of the system. The dynamic equations of the n-degree-of-freedom manipulator are as follows [16]:

$$\begin{array}{c}\hfill \begin{array}{c}M\left(\psi \right)\ddot{\psi }+C(\psi ,\dot{\psi })\dot{\psi }+G\left(\psi \right)=D\left(\tau \right)\hfill \end{array}\end{array}$$

(1)

The variables $\psi$, $\dot{\psi }$, and $\ddot{\psi }$∈$R^n$ correspond to the joint position, joint velocity, and joint acceleration, repectively. The inertia matrix is represented by $M\left(\psi \right)$∈$R^{n\times n}$, the centripetal and Coriolis torque is represented by $C(\psi ,\dot{\psi })$∈$R^{n\times n}$, the gravitational force is denoted by $G\left(\psi \right)$∈$R^n$, $\tau$∈$R^n$ represents the joint torque, and $D\left(\tau \right)$ is the deadzone function for the control input $\tau$. The deadzone $D\left({\tau }_i\right)$ is expressed as

$$D\left({\tau }_i\right)=\left\{\begin{array}{cc}{h}_r\left({\tau }_i-b_r\right)\hfill & {\tau }_i\ge b_r\hfill \\ 0\hfill & b_l<{\tau }_i<b_r\hfill \\ h_l\left({\tau }_i-b_l\right)\hfill & {\tau }_i\le b_l\hfill \end{array}\right.$$

(2)

where $h_r$ and $h_l$ are the known nonlinear functions and $b_r>0$ and $b_l<0$ are known constants. This paper investigates the Phantom Omni robot, as illustrated in Figure 1a. The schematic diagram, showing the reference frames used in the dynamics, is shown in Figure 1b. The robot is a three-DoF manipulator, and its dynamics are described by Equation (1), with the matrices M, C, and G, as given in [17].

$$\begin{array}{ccc}\hfill M& =\left[\begin{array}{ccc}{m}_{11}& 0& 0\\ 0& m_{22}& m_{23}\\ 0& m_{32}& m_{33}\end{array}\right],G=\left[\begin{array}{c}0\\ g{k}_5{c}_2+g{k}_6{c}_{23}\\ g{k}_6{c}_{23}\end{array}\right],C\hfill & \hfill =\left[\begin{array}{ccc}-a_1{\dot{\psi }}_2& -a_1{\dot{\psi }}_1& -a_2{\dot{\psi }}_1\\ a_1{\dot{\psi }}_1& -a_3{\dot{\psi }}_3& -a_3\left({\dot{\psi }}_2+{\dot{\psi }}_3\right)\\ a_2{\dot{\psi }}_1& a_3{\dot{\psi }}_2& 0\end{array}\right]\end{array}$$

(3)

where $m_{11}=k_1+k_2{c}_2^2+k_3{c}_{23}^2+2k_4{c}_2{c}_{23},m_{22}=k_2+k_3+2k_4{c}_3,m_{23}=k_3+k_4{c}_3,$ $m_{32}=m_{23},m_{33}=k_3,a_1=k_2{c}_2{s}_2+k_3{c}_{23}{s}_{23}+k_4{c}_{2\times 23},a_2=k_3{c}_{23}{s}_{23}+k_4{c}_2{s}_{23},$ $a_3=k_4{s}_3,s_i=\sin \left({\psi }_i\right),s_{23}=sin\left({\psi }_2+{\psi }_3\right),c_i=cos\left({\psi }_i\right),c_{23}=cos\left({\psi }_2+{\psi }_3\right),$ $c_{2\times 23}=cos\left(2{\psi }_2+{\psi }_3\right)$.

3. Preliminaries

3.1. Reinforcement Learning and Soft Actor-Critic

In reinforcement learning (RL), an agent aims to optimize its decision-making process to accumulate maximum reward by interacting with the environment. Figure 2 shows how an agent learns during reinforcement learning. In this framework, the agent operates in discrete time steps. At each step t, the agent perceives the current state $s_t$, selects an action $a_t$, receives a corresponding reward $r_t$, and transitions to a new state $s_{t+1}$. The objective is to derive a policy $\pi \left(a\right|s)$ that maximizes the expected return, expressed as $R={\sum }_{t=0}^T{\gamma }^t{r}_t$, with $\gamma$ being the discount factor.

SAC is an RL algorithm designed for continuous action spaces. It combines the advantages of maximum entropy RL with the stability of actor-critic methods. The policy (actor) update in the SAC algorithm seeks to maximize both the expected reward and the entropy of the policy, encouraging exploration. Built upon the actor-critic framework and maximum entropy RL theory, the SAC algorithm focuses on optimizing the policy expectations along with the standard RL return [18].

$${\mathsf{\Pi }}^{★}=\underset{\mathsf{\Pi }}{argmax}\sum _t^{T}E[r(s_t,a_t)+\alpha \mathcal{H}\left(\pi (·|s_t)\right)]$$

(4)

where $\mathcal{H}\left(\pi (·|s_t)\right)$ represents the entropy of the policy.

The SAC algorithm involves five networks: two soft Q-value networks, two target soft Q-value networks, and a policy network. The Bellman equation is applied to update the soft Q-value networks, where the update target for a Q-value network ${\mathsf{\Theta }}_i$ is specified in [19]:

$$L_Q\left({\mathsf{\Theta }}_i\right)=E\left[{\displaystyle \frac{1}{2}}{\left(Q_{{\mathsf{\Theta }}_i}(s,a)-y(r,s^{\prime },d)\right)}^2\right]$$

(5)

SAC ensures stable training by using the minimum value from the two target Q-value networks as the target. The Q-value networks are then updated using stochastic gradient descent:

$${\nabla }_{{\mathsf{\Theta }}_i}L\left({\mathsf{\Theta }}_i\right)={\nabla }_{{\mathsf{\Theta }}_i}{Q}_{{\mathsf{\Theta }}_i}(s,a)\left[{\left(Q_{{\mathsf{\Theta }}_i}(s,a)-y(r,s^{\prime },d)\right)}^2\right]$$

(6)

The policy network ${\mathsf{\Pi }}_{\varphi }\left(a\right|s)$ is updated by maximizing the expected value of the Q-value and the entropy term under the current policy:

$$J_{\mathsf{\Pi }}\left(\varphi \right)=E_{\left(s\right)\sim D}\left[E_{a\sim {\mathsf{\Pi }}_{\varphi }}\left[\alpha log{\mathsf{\Pi }}_{\varphi }\left(a\right|s)-Q_{{\mathsf{\Theta }}_i}(s,a)\right]\right]$$

(7)

Stochastic gradient descent is then used to update the policy network:

$$\begin{array}{c}\hfill {\nabla }_{\varphi }{J}_{\mathsf{\Pi }}\left(\varphi \right)={\nabla }_{\varphi }\alpha log{\mathsf{\Pi }}_{\varphi }\left(a\right|s)+({\nabla }_{\varphi }\alpha log{\mathsf{\Pi }}_{\varphi }\left(a\right|s)-\nabla Q_{{\mathsf{\Theta }}_i}(s,a))\end{array}$$

(8)

3.2. Long Short-Term Memory

LSTM is a variant of recurrent neural networks (RNNs) specifically designed for modeling sequential data and capturing long-term dependencies. The forget gate governs the removal of information from the cell state. The forget gate of LSTM determines what information to discard from the cell state $C_{t-1}$ [20]:

$$f_t=\phi \left(W_f^{\prime }·\left[h_{t-1},x_t\right]+b_f^{\prime }\right)$$

(9)

The input gate governs the decision of what new information to store in the cell state:

$$\begin{array}{c}{ß}_t=\phi \left(W_i^{\prime }·\left[h_{t-1},x_t\right]+b_i^{\prime }\right)\\ {\tilde{C}}_t=tanh\left(W_C^{\prime }·\left[h_{t-1},x_t\right]+b_C^{\prime }\right)\end{array}$$

(10)

where input gate ${ß}_t$ scales the candidate values ${\tilde{C}}_t$.

The output gate governs the output of the LSTM cell. It merges the current cell state with the activation of the output gate to produce the final hidden state:

$$\begin{array}{c}{\eta }_t=\phi \left(W_o^{\prime }·\left[h_{t-1},x_t\right]+b_o^{\prime }\right)\\ {\widehat{h}}_t={\eta }_t·tanh\left(C_t^{\prime }\right)\end{array}$$

(11)

The symbol $\phi$ mentioned above refers to the sigmoid function, while the weight matrix and bias term are represented by W and b, respectively. The combined operation of these gates enables the LSTM unit to learn long-term dependencies, which is crucial for sequential data processing.

3.3. Attention Mechanism

Attention mechanisms have become a critical component in sequence modeling, enabling models to focus on the most relevant parts of the input sequence when producing output. The attention mechanism dynamically assigns different weights to different input elements based on their relevance to the current state or prediction, improving the model’s ability to capture long-range dependencies and handle variable-length sequences.

In the context of sequence-to-sequence models, attention mechanisms allow the model to focus on different parts of the input sequence at each time step.

3.4. Random Network Distillation

Random network distillation (RND) is an RL exploration method aimed at improving an agent’s ability to investigate unfamiliar environments, thus boosting learning efficiency. Similar to the Intrinsic Curiosity Module (ICM) algorithm, RND uses a fixed target network, which is typically a randomly initialized neural network, to forecast changes in environmental states. The next state observation $s_{t+1}$ is input into this target network to produce the output $f\left(s_{t+1}\right)$. The prediction network shares the same architecture and input as the target network, with its output being $\widehat{f}\left(s_{t+1}\right)$, the state encoding $f\left(s_{t+1}\right)$ from the target network. The prediction error $ϵ$ is computed as follows [21]:

$$ϵ=\parallel \widehat{f}\left(s_{t+1}\right)-f\left(s_{t+1}\right){\parallel }^2$$

(12)

Serving as an internal reward, the prediction error $ϵ$ guides the agent’s exploration. RND incentivizes the agent to explore states where the target network’s predictions are less accurate, promoting effective learning and aiding exploration in intricate environments. The RND is widely employed in various domains, including gameplay, robot control, and autonomous driving, demonstrating its effectiveness in enhancing exploration and improving agent performance.

4. Controller Design

4.1. Design of System Input and Output

In DRL, a network of actors generates output actions based on the current input state. In the context of trajectory tracking of the manipulator in the task space, the joint angles are crucial [22]. The goal of the controller is to choose appropriate actions from the real-time state information of the robot arm to ensure superior tracking control. The state space is outlined as follows:

$$s={\left[\begin{array}{cccc}\psi \hfill & \dot{\psi }\hfill & {\psi }_d\hfill & \dot{{\psi }_d}\hfill \end{array}\right]}^T$$

(13)

where ${\psi }_d$ and $\dot{{\psi }_d}$ denote the target angular position and target angular velocity. Additionally, to control the swing amplitude during target curve tracking, saturation constraints are enforced on the manipulator joints. The specific limitations are

$$\psi =\left\{\begin{array}{cc}{\psi }_{min}\hfill & ,\mathrm{if}{\psi }_i\le {\psi }_{min}\hfill \\ {\psi }_i\hfill & ,\mathrm{if}{\psi }_{min}<{\psi }_i<{\psi }_{max},i=1,2,\dots ,n\hfill \\ {\psi }_{max}\hfill & ,\mathrm{if}{\psi }_i\ge {\psi }_{max}\hfill \end{array}\right.$$

(14)

where ${\psi }_{max}=\pi$ and ${\psi }_{min}=-\pi$.

In RL tasks, the reward function is a key component. In the context of robotic manipulator control, the tracking error $e\left(t\right)$ is the primary variable. The reward function can be written as

$$r_t=-\parallel e_i\left(t\right)\parallel$$

(15)

where $e_i\left(t\right)$ = $\psi \left(t\right)$ − ${{\psi }_d}_i\left(t\right)$.

4.2. Network Architecture

Inspired by [14], this article combines the SAC algorithm with an LSTM network. SAC-LSTM is based on the actor–critic architecture, comprising a policy and Q-value networks. The policy network consists of an LSTM layer, attention mechanism layers, fully connected layers, and output layers. The policy network is shown in Figure 3a. The framework employs an attention mechanism to dynamically focus on relevant features in the LSTM output. In [23], the last hidden state of LSTM is used as the input of the fully connected layer. In the trajectory tracking task of this paper for manipulator, the state sequence at each time step is fed into the LSTM, which outputs a hidden state h for each time step. This attention mechanism layer computes three linear transformations of h to produce the query, key, and value matrices. The query and key matrices are used to calculate attention weights through a softmax function applied to their dot product, which allows the model to focus on different parts of the sequence by assigning higher weights to more relevant features. These attention weights are then used to produce a weighted sum of the value matrix, effectively generating a context vector that emphasizes important temporal features. These hidden states h are passed to fully connected layers to generate the Gaussian distribution parameters for the actions (torques): the mean (using the tanh activation function) and the standard deviation (using the softplus activation function with a small constant added to avoid numerical issues). During training, actions are sampled from this distribution to ensure they are random and exploratory. The final actions are scaled by the action bounds after passing through the tanh function, and their log probabilities are computed. As the network converges, the variance of the action distribution gradually decreases.

The Q-value network consists of fully connected layers and an output layer, as shown in Figure 3b. It takes the concatenation of the state sequence and the action as input. First, the concatenated input is fed into the first fully connected layer and processed by the ReLU activation function. The result is then passed to the second fully connected layer, also processed by the ReLU activation function. Finally, the output layer produces a single value, $Q(s,a)$, representing the expected return for the given state and action.

4.3. SLR Algorithm Design

For improving the ability and efficiency of robot manipulator control, the RND method is introduced and the SLR algorithm is proposed. The RND can prompt the robotic arm agent to more actively explore unpredictable and unknown environmental states, leading to the discovery of superior policies during the training of arm control. Therefore, combining the RND method with the SAC-LSTM algorithm can obtain more stable and effective control policies.

The RND target and predictor networks consist of a fully connected layer and an output layer. Since the weights of the target network remain unchanged, the prediction network can focus on the intrinsic representation of the learning state during the training process, thereby enhancing both the stability and efficiency of the learning process [24]. The difference in output values between the two networks acts as an internal reward for the robot manipulator environment. Then, the total reward for RL algorithm training is defined as the sum of the intrinsic and extrinsic rewards, formed by the linear combination of the values in Equations (12) and (15). It is expressed as follows:

$$r_{total}=\sum _{i=1}^n-w_i|e_i\left(t\right)|+||f_{i+1}-f_{i+1}^{\prime }{\left|\right|}^2$$

(16)

The $r_{total}$ is then used in Equation (7) to update the actor network. This encourages the agent to explore unpredictable and unknown environmental states more actively, enhancing both learning efficiency and performance. The algorithm pseudocode is shown in Algorithm 1, and the algorithm flow chart is shown in Figure 4.

Algorithm 1 SLR.

Input:  The parameter of policy network $\varphi$, the parameters of Q Network ${\mathsf{\Theta }}_1$, ${\mathsf{\Theta }}_2$ Input:  target network weights ${\widehat{\mathsf{\Theta }}}_1\leftarrow {\mathsf{\Theta }}_1,{\overline{\mathsf{\Theta }}}_2\leftarrow {\mathsf{\Theta }}_2$ Output:  Optimazed parameters   1: for each iteration do   2:       for each step do   3:             Get $\mathbf{s}$ and select $\mathbf{a}\sim {\pi }_{\varphi }(.\mid {\mathbf{s}}_t)$;   4:             Excute $\mathbf{a}$ in the environment;   5:             Get ${\mathbf{s}}_{\mathbf{t}+\mathbf{1}}$, reward r, and done singal $\mathbf{d}$   6:                to indicate whether ${\mathbf{s}}_{\mathbf{t}+\mathbf{1}}$ is terminal state;   7:             Calculate internal reward $r^{\prime }=\left|\right|f_{i+1}-f_{i+1}^{\prime }{\left|\right|}^2$   8:             Calculate augmented reward $r_{total}=r+r^{\prime }$   9:             Store ($\mathbf{s}$, $\mathbf{a}$, $r_{total}$, ${\mathbf{s}}^{\prime }$, $\mathbf{d}$) in replay buffer $\mathcal{D}$; 10:             if ${\mathbf{s}}^{\prime }$ is terminal then 11:                  Reset environment; 12:             end if 13:       end for 14:       for each gradient step do 15:             Sample data B = ($\mathbf{s}$, $\mathbf{a}$, $\mathbf{r}$, ${\mathbf{s}}^{\prime }$, $\mathbf{d}$) from $\mathcal{D}$; 16:             Compute targets Q networks following Equation (5) 17:             Update Q network parameters using 18:                gradient descent Equation (6) 19:             Update policy network parameters using 20:                gradient descent Equation (8) 21:             Update the parameters of target network using 22:                ${\varphi }_{\mathrm{targ},i}\leftarrow \rho {\varphi }_{\mathrm{targ},i}+(1-\rho ){\varphi }_i$ for $i=1,2$ 23:       end for 24: end for

5. Simulation

This section presents simulation experiments conducted using the simulation environment from Section 2 to assess the performance of the method. To evaluate the performance of our SLR algorithm, we selected the SAC, SAC-RND, and SAC-LSTM methods as benchmarks. At the same time, the anti-interference ability of the four methods was assessed. All experiments were performed using the Phantom Omni robotic manipulator. Considering the availability of laboratory hardware, a workstation was used to create a virtual robotic manipulator system. The initial state values were defined as ${\psi }_1\left(0\right)=0.23$, ${\psi }_2\left(0\right)=-0.07$, and ${\psi }_3\left(0\right)=-0.022$. The total number of steps was 3000, with a step size of 0.01 s per step.

5.1. Training Performance

The training performance of the SAC, SAC-RND, SAC-LSTM, and SLR methods wass assessed in this experiment. Figure 5 shows the trends in cumulative reward values for the four algorithms, along with the variance and standard deviation of the cumulative reward. The results suggest that the reward of the SLR agent converges around the 50th episode, and the learning curve continues smoothly after that point. In contrast, the SAC-LSTM and SAC-RND agents only converge after the 100th episode, with their learning curves continuing to fluctuate even after convergence. Throughout the training process, the reward curve of the SAC algorithm exhibits fluctuations. Meanwhile, the variance and standard deviation of the SLR agent stabilize after the 75th episode, whereas those of the SAC, SAC-RND, and SAC-LSTM agents continue to fluctuate significantly.

An ablation experiment was conducted to evaluate the impact of the LSTM network with the attention mechanism and the RND method enhancement. When not combined with the LSTM network, the convergence speed of the SAC-RND algorithm is higher than that of the SAC algorithm, but the stability during training is not effectively alleviated. In addition, the reward curve of the SAC-LSTM algorithm is very smooth, but its reward value after convergence is not as high as that of SAC-RND. However, the stability and convergence speed of the SLR algorithm proposed in this paper are better than those of other benchmark algorithms during training. This result shows that the LSTM network with an attention mechanism is key to improving stability, while RND is essential for enhancing exploration. Combining them for manipulator control enables us to enhance the efficiency and robustness of the algorithm.

5.2. Control Performance

In this experiment, the performance of SLR in tracking control was compared to that of SAC, SAC-RND, and SAC-LSTM. In Figure 6 and Figure 7, the tracking trajectories, tracking errors, and torque variations of the joints are presented for SAC, SAC-LSTM, SAC-RND, and SLR during control tasks in both environments: one without deadzones and one with deadzones.

As shown in Figure 6a–c, the control trajectories in the three joints are presented for the environment without deadzones. The tracking errors of all joints are shown in Figure 6d–f. These results indicate that SLR outperforms the baseline methods in tracking control performance, with smaller deviations between the reference and actual positions. The absence of deadzones allows for more precise tracking and reduces delays in response, although SLR still demonstrates superior tracking accuracy compared to SAC, SAC-LSTM, and SAC-RND.

In contrast, in the environment with deadzones, as shown in Figure 7a–c, the control trajectories of the three joints exhibit similar trends, but with slightly larger deviations from the desired path. The tracking errors for all joints in the deadzone environment are shown in Figure 7d–f. SLR still outperforms the baseline methods, although the tracking errors are higher compared to the no-deadzone scenario due to the inherent delays and nonlinearities introduced by the deadzones.

The torque trends for the joints in the no-deadzone environment are shown in Figure 6g–i. These results demonstrate that the torque of all joints is smaller when controlled by SLR compared to SAC, SAC-LSTM, and SAC-RND, highlighting the efficiency of SLR in minimizing energy consumption and mechanical stress. In the deadzone environment, as illustrated in Figure 7g–i, the torque variations are still lower for SLR but exhibit some fluctuation due to the deadzone effects, although SLR maintains a clear advantage in reducing torque compared to the other algorithms.

As demonstrated by this experiment, the SLR algorithm leverages the strengths of RND and LSTM to quickly learn stable control policies, resulting in excellent control ability with reduced torque. The average absolute tracking error (AAT), average absolute tracking error (AAE), and root-mean-squared error (RMSE) are summarized in

Table 1. AAE, AAT, and RMSE of the SAC, SAC-RND, SAC-LSTM, and SLR algorithms in the no-deadzone environment.

	Algorithms	SAC	SAC-RND	SAC-LSTM	SLR
Joint
joint1	AAE	0.0141	0.0076	0.0072	0.0055
	AAT	11.07	12.35	10.22	9.45
	RMSE	0.0223	0.0123	0.0132	0.0097
joint2	AAE	0.0053	0.0059	0.052	0.0045
	AAT	5.732	10.23	6.702	3.612
	RMSE	0.0349	0.0108	0.0118	0.0131
joint3	AAE	0.0112	0.0123	0.0092	0.0063
	AAT	8.713	7.349	4.032	1.692
	RMSE	0.0665	0.0221	0.0184	0.0151

and

Table 2. AAE, AAT, and RMSE of the SAC, SAC-RND, SAC-LSTM, and SLR algorithms in the dead zone environment.

	Algorithms	SAC	SAC-RND	SAC-LSTM	SLR
Joint
joint1	AAE	0.0152	0.0088	0.0081	0.0064
	AAT	12.39	13.86	11.26	10.98
	RMSE	0.0241	0.014	0.0145	0.0113
joint2	AAE	0.0059	0.0067	0.058	0.0051
	AAT	6.663	11.26	7.468	3.781
	RMSE	0.0393	0.0114	0.0126	0.0144
joint3	AAE	0.013	0.0138	0.01	0.0076
	AAT	10.08	8.377	4.215	1.941
	RMSE	0.0726	0.0248	0.0202	0.0172

for both environments, with the corresponding formulas for AAE, AAT, and RMSE as follows:

$$\mathrm{AAE}={\displaystyle \frac{1}{T}}\sum _{i=1}^T\left|e_i\right|i=1,2,\dots ,n$$

(17)

$$\mathrm{AAT}={\displaystyle \frac{1}{T}}\sum _{i=1}^T\left|{\tau }_i\right|i=1,2,\dots ,n$$

(18)

$$\mathrm{RMSE}=\sqrt{{\displaystyle \frac{1}{T}}\sum _{i=1}^T{e}_i^2}i=1,2,\dots ,n$$

(19)

where T is the total time step.

The results for both environments highlight the robustness of the SLR algorithm, particularly in terms of minimizing tracking errors and joint torque, showcasing its potential for precise and energy-efficient robotic control.

6. Conclusions

This paper proposes an SLR (SAC-LSTM-RND) algorithm to train robotic manipulators to perform tracking control tasks. According to the characteristics of the robotic manipulator system, the SAC-LSTM algorithm is implemented. LSTM is used to capture the timing information of the curve trajectory and the attention mechanism can dynamically focus on critical features, thereby improving the adaptability of the controller to time-varying curves. SAC-LSTM solves the instability when training with only the SAC algorithm. To enhance the exploration ability of the robot arm agent, the RND method is added to the algorithm. RND prompts the robot arm agent to explore uncertain states of the environment more proactively, thereby obtaining superior policies for robotic manipulator control. The results demonstrate that, under the same conditions, the algorithm proposed in this paper exhibits superior learning and control capabilities. In addition, the algorithm proposed in this paper has only been verified in a single task environment. Future work can be extended to more diverse practical tasks, including multi-task learning, collaborative control, and other scenarios. This will not only verify the effectiveness of the algorithm in more complex environments, but also further promote the development of the algorithm towards a general control strategy.