Trajectory Control of An Articulated Robot Based on Direct Reinforcement Learning

Abstract

Reinforcement Learning (RL) is gaining much research attention because it allows the system to learn from interacting with the environment. Yet, with all these successful applications, the application of RL in direct joint torque control without the help of an underlining dynamic model is not reported in the literature. This study presents a split network structure that enables successful training of RL to learn the direct torque control for trajectory following a six-axis articulated robot without prior knowledge of the dynamic robot model. The training took a very long time to converge. However, we were able to show the successful control of four different trajectories without needing an accurate dynamics model and complex inverse kinematics computation. To show the RL-based control’s effectiveness, we also compare the RL control with the Model Predictive Control (MPC), another popular trajectory control method. Our results show that while the MPC achieves smoother and more accurate control, it does not automatically treat the singularity. In addition, it requires complex inverse dynamics calculations. On the other hand, the RL controller instinctively avoided the violent action around the singularities.

1. Introduction

Reinforcement Learning allows the system to learn by interacting with the environment. There is no need for a large number of labeled samples, and the system learns as it accumulates experience. Since its introduction, RL has shown great potential in many applications.

This study addresses the direct application of Reinforcement Learning (RL) to the joint force control of an articulated robot to follow pre-specified trajectories. The RL controller uses a neural network to compute the required joint torque. It requires no knowledge of an underlying dynamic robot model and needs no complex inverse kinematics calculation. From the application point of view, a neural network-based robot arm controller is desirable. However, very few available RL robot control examples have addressed the direct torque control of articulated robots. Searching through the literature, one finds that most available results dealt only with the kinematic trajectory planning and left the torque control problem to the controller that came with the robot. Many related results have based their control on model-based inverse dynamic cancellation and used RL to learn the control parameters [1,2]. The others that did address direct joint torque were for SCARA robots with only two links [2,3]. Considering how humans learn to control their limbs, it is still desirable to know if one can train the robot arm the way we train ourselves to use our arm.

Learning control has been around for a long time. In 1995, Nuttin and Van Brussel proposed a learning controller to increase the insertion speed in consecutive peg-into-hole operations; the learning controller could perform the same without increasing the contact force level [4]. In 1997, Schaal examined the types of learning problems that benefit from demonstration [5]. He compared the performance of various learning algorithms on an anthropomorphic robotic arm. In 2006, Peters and Schaal demonstrated the efficiency of Reinforcement Learning in teaching a SARCOS Master Arm to hit a baseball [6]. In 2007, he also performed the Reinforcement Learning approach on the anthropomorphic robotic arm simulation in operational space control [7]. In 2010, Kober et al. explained the technique of learning mappings from circumstances to meta-parameters by using Reinforcement Learning. He showed two robot movement applications: throwing darts and table tennis hitting [8]. In 2016, Gu et al. showed that a Deep Reinforcement Learning algorithm based on the off-policy training of Deep Q-Functions could scale to complex 3D manipulation tasks for a robot [9]. In 2017, Liu et al. proposed an RL learning control for the robotic insertion task [10]. In 2018, Luo et al. used Guided Policy Search in the peg-in-hole insertion in a deformable hole [1].

Many RL approaches are available. The Deep Deterministic Policy Gradients (DDPG) is a state-of-the-art algorithm to simultaneously train the Deep Q Network and the Deep Policy Network. DDPG achieves good performance in continuous control problems by integrating the Deterministic Policy Gradient algorithm. From a comparative viewpoint, the difference between the Stochastic- and Deterministic Policy Gradient is that the Stochastic Policy Gradient integrates over both state and action spaces. In contrast, the Deterministic Policy Gradient only integrates over the state space. Consequently, the Deterministic Policy Gradient requires fewer samples than the Stochastic Policy Gradient, especially if the action space has few dimensions [11]. Survey papers at various stages of development on RL are already available. Kormushev provided a summary describing the RL application in the real world industry [12], and Wang et al. published a review of recent theoretical development [13]. There is also the convergence issue with the training algorithm. When Silver et al. first proposed the deterministic policy gradient (DPG) algorithm [11], they required a linear compatible function as the state-action value function. The DDPG proposed by Lillicrap et al. [14] uses a deep neural network (DNN) as a function approximator. The nonlinear approximator introduces a risk of trapping in the local optimal. Many literature results have shown that the main issue is the compatibility between the actor and critic networks. To avoid this situation, DDPG introduces the noisy state-action value function. Many research efforts have been devoted to exploring ways to improve convergence [15,16,17,18,19,20,21,22]. There are also alternative approaches to DDPG for continuous-time reinforcement learning. Ghavamzadeh and Mahadevan [23] developed a generalization of the MAXQ for continuous-time semi-Markov decision processes. Tiganj et al. [24] developed a logarithmically-compressed time scale model for the future to establish a scale-invariant timeline to overcome the discretization problem. Research by Jiao and Oh [25] proposed using separate threads to execute the “behavior” and the “update policy.” In this research, we adopted the method proposed by [14] to add noise to the actor policy to ensure value function compatibility.

This study used a HIWIN RA605 robotic arm for the target experiment. We defined the simulation’s target trajectory, reward function, policy, and actor-critic network structures. After many trials, it became clear that the direct application of RL to multi-link articulated robot trajectory control without an underlining model is more complicated than one expects. A recent study directly applied the deep learning controller to emulate the inverse dynamics action [26]. They handled the nonlinear behavior by applying the learning control on the separate axis with all the other axes fixed. However, they still used model-based supervised learning. In addition, their effort to train all the axes can be too great for practical implementation. Instead, our study based the training on the trajectory without needing a dynamic model. One can therefore train the RL network along a specific path. We describe the proposed network structure and the result of the training.

This study also compared the control with the Model Predictive Control (MPC), another popular control method, to demonstrate the control effectiveness. MPC based the design on a system model. Instead of conventional feedback, it determines the control action by optimizing a performance index over a finite time horizon [27]. Many previous researchers have proposed different MPC methods. In 2014, Nikdel designed an MPC controller to control a single-degree-of-freedom shape memory alloy (SMA) actuated rotary robotic arm to track the desired joint angle [28]. MPC discovered practices in dynamic and unpredictable environments such as chemical plants and oil refineries [29], as well as power system balancing [30]. More recently, the MPC has also begun to find applications in the control of autonomous vehicles [31] and robotic trajectory control [32,33,34]. In 2016, Best et al. applied MPC to robot joint control. The dynamic model-based control of a soft robot shows significant improvement over the regular controller [35]. In 2017, Lunni et al. applied a Nonlinear Model Predictive Controller (NMPC) to an aerial robotic arm for tracking desired 3D trajectories. They demonstrated online use of the NMPC controller with limited computational power [36]. In 2018, Guechi et al. proposed a combination of MPC and feedback linearization control for a two-link robotic arm [37,38]. In the same year, Car et al. applied an MPC-based position control to an unmanned aerial vehicle (UAV). The onboard computer running linear MPC-based position control communicates through a serial port with the low-level altitude controller [39].

This paper describes using RL to control a HIWIN RA605 robotic arm. The task is for the robot arm to follow a desired trajectory while maintaining a specific attitude for the end effector. We propose a split network structure that allows for the direct application of RL to robot trajectory control. We also compare the control performance to a Twin Delayed Deep Deterministic Policy Gradient (TD3) network and an MPC. The results show that the proposed RL structure can achieve articulated robot control without the help of an accurate robot model. The RL trajectory is oscillatory but instinctively avoids the singularity and requires no background model. The behavior is similar to a human holding out their hand. Although the TD3 controller still converges, it oscillates around a noticeable offset. The MPC is smooth but requires a system model and additional treatments to avoid singularities.

2. Theoretical Background

2.1. Deep Deterministic Policy Gradient (DDPG)

The popular DDPG was first used in this study to train the RL network. The standard reinforcement learning algorithm considers the problem of an agent interacting with an environment $E$. The policy $\pi$ for the agent’s action $a$ at state $\mathit{s}$ is represented by a probability distribution $\pi :\mathcal{S}\to \mathcal{P}\left(\mathcal{A}\right)$, where $\mathcal{S}$ represents the state space and $\mathcal{A}$ represents the action space. Assuming that the actions can be parameterized with parameters $\mathit{\theta }$; the parameterized policy becomes ${\pi }_{\mathit{\theta }}\left(\mathit{s}\right)$ which selects the action $a$ in state $\mathit{s}$ with parameters $\mathit{\theta }$ by interacting with the environment. At each time step $t$, the agent receives a reward $r$ for its action, and the return from a state is defined by the sum of the discounted future reward $R={{\displaystyle \sum }}_t{\gamma }^{i-1}r\left({\mathit{s}}_t,a_t\right),\gamma \in \left[0,1\right]$. As a result, the expected return, or the value function, for the action $a={\pi }_{\mathit{\theta }}\left(\mathit{s}\right)$ becomes $Q^{\pi }\left({\mathit{s}}_t,{\pi }_{\mathit{\theta }}\left({\mathit{s}}_t\right)\right)={\mathbb{E}}_{r,\mathit{s}~E,a~\pi }[R|{\mathit{s}}_t,a_t]$. The goal of the agent is to maximize the value function $Q$ by updating ${\pi }_{\mathit{\theta }}\left({\mathit{s}}_t\right)$ and $Q^{\pi }\left({\mathit{s}}_t,{\pi }_{\mathit{\theta }}\left({\mathit{s}}_t\right)\right)$. To allow an iterative learning process, one can express the value function $Q$ in a recursive form: $Q^{\pi }\left({\mathit{s}}_t,a_t\right)={\mathbb{E}}_{r_t,s_{t+1}~E}\left[r\left({\mathit{s}}_t,a_t\right)+\gamma {\mathbb{E}}_{a_{t+1}~\pi }\left[Q^{\pi }\left({\mathrm{s}}_{t+1}, a_{t+1}\right)\right]\right]$. One can omit the inner expectation if one chooses a deterministic policy as $\mu :\mathcal{S}\to \mathcal{A}$, and reduces the value function to an expression that depends only on the environment such as $Q^{\mu }\left({\mathit{s}}_t,a_t\right)={\mathbb{E}}_{r_t,s_{t+1}~E}\left[r\left({\mathit{s}}_t,a_t\right)+\gamma Q^{\mu }({\mathit{s}}_{t+1},\mu \left({\mathit{s}}_{t+1}\right)\right]$. It is now possible to use an off-policy algorithm to learn $Q^{\mu }$ with a greedy policy $\mu \left(\mathit{s}\right)=\arg \underset{\mathrm{a}}{\max }Q\left(\mathit{s},a\right)$.

The standard RL algorithm is not directly applicable to continuous action space due to the need for optimization at every time step. DDPG thus resolves into an actor-critic model structure. The actor specifies the deterministic policy by mapping the states to a specific action. The critic then learns the value function using the Bellman equation. The policy gradient of the expected return can be expressed as:

$$\begin{gathered} {\nabla }_{{\mathit{\theta }}^{\mu }}J\approx {\mathbb{E}}_{{\mathit{s}}_t~{\rho }^{\beta }}\left[{\nabla }_{{\mathit{\theta }}^{\mu }}Q\left(\mathit{s},a|{\mathit{\theta }}^Q\right){\left.\right|}_{\mathit{s}={\mathit{s}}_t,a=\mu \left({\mathit{s}}_t|{\mathit{\theta }}^{\mu }\right)}\right] \\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}={\mathbb{E}}_{{\mathit{s}}_t~{\rho }^{\beta }}\left[{\nabla }_{a}Q\left(\mathit{s},a|{\mathit{\theta }}^Q\right){\left.\right|}_{\mathit{s}={\mathit{s}}_t,a=\mu \left({\mathit{s}}_t\right)}{\nabla }_{{\mathit{\theta }}_{\mu }}\mu \left(\mathit{s}|{\mathit{\theta }}^{\mu }\right){\left.\right|}_{\mathit{s}={\mathit{s}}_t}\right]. \end{gathered}$$

(1)

The basic algorithm for DDPG is shown in Algorithm 1.

Algorithm 1: DDPG

• Initialize $\mathit{\theta }$ and $\mathit{\varphi }$ arbitrarily, set $\widehat{\mathit{\theta }}=\mathit{\theta }$ and $\widehat{\mathit{\varphi }}=\mathit{\varphi }$, iterate until they converge.
• For every episode:
  • For every time step $t$:
    • Given a state ${\mathit{s}}_t$, take action $a_t$ based on policy $\pi$;
    • Obtain reward $r_t$ and the new state ${\mathit{s}}_{t+1}$;
    • Store all the (${\mathit{s}}_t$,$a_t$,$r_t$,${\mathit{s}}_{t+1}$) into the buffer.
  •  
    • Sample a batch of (${\mathit{s}}_i$,$a_i$,$r_i$,${\mathit{s}}_{i+1}$) from the buffer;
    • Set the cost function $C\left(\mathit{\theta }\right)={\displaystyle \sum }_i{\left[r_i+\gamma \widehat{Q}\left({\mathit{s}}_{i+1},\widehat{\pi }\left({\mathit{s}}_{i+1}\right)\right)-Q\left({\mathit{s}}_i,\pi \left({\mathit{s}}_i\right)\right)\right]}^2$;
    • Update the parameters of $Q$, which is $\mathit{\theta }$, to minimize $C\left(\mathit{\theta }\right)$.

$${\mathit{\theta }}^{*}=\mathit{\theta }-\eta \frac{\partial C\left(\mathit{\theta }\right)}{\partial \mathit{\theta }}$$

(2)

Set the value function:

$$V\left(\mathit{\varphi }\right)={\displaystyle \sum }_{i}Q\left({\mathit{s}}_i,\pi \left({\mathit{s}}_i\right)\right)$$

(3)

Update the parameters of $\pi$, which is $\mathit{\varphi }$, to maximize $V\left(\mathit{\varphi }\right)$.

$${\mathit{\varphi }}^{*}=\mathit{\varphi }+\lambda \frac{\partial V\left(\mathit{\varphi }\right)}{\partial \mathit{\varphi }}=\mathit{\varphi }+\lambda {\displaystyle \sum }_i\frac{\partial Q\left({\mathit{s}}_i,\pi \left({\mathit{s}}_i\right)\right)}{\partial \mathit{\varphi }}=\mathit{\varphi }+\lambda {\displaystyle \sum }_i\frac{\partial Q\left({\mathit{s}}_i,\pi \left({\mathit{s}}_i\right)\right)}{\partial \mathit{\varphi }} =\mathit{\varphi }+\lambda {\displaystyle \sum }_i\frac{\partial Q\left({\mathit{s}}_i,\pi \left({\mathit{s}}_i\right)\right)}{\partial T\left({\mathit{s}}_i\right)}\frac{\partial \pi \left({\mathit{s}}_i\right)}{\partial \mathit{\varphi }}$$

(4)

Update $\widehat{\mathit{\theta }}=\tau {\mathit{\theta }}^{*}+\left(1-\tau \right)\mathit{\theta }$ and $\widehat{\mathit{\varphi }}=\tau {\mathit{\varphi }}^{*}+\left(1-\tau \right)\mathit{\varphi }$, where $\tau$ is a factor between 0 and 1.

2.2. The Modified DDPG Method

As mentioned before, the algorithm in the previous subsection adopted the greedy policy to optimize $a_t$ for each time step and is unsuitable for applying to continuous-time robotic motion control. Therefore, modifying the algorithm into an applicable actor-critic algorithm is necessary.

In the DDPG algorithm, $\pi \left({\mathit{s}}_t\right)$ is used to represent the decision-making policy that determines the corresponding action in the state ${\mathit{s}}_t$. The decision of the action $a_t$ by $\pi \left({\mathit{s}}_t\right)$ does not result directly from the information of the state ${\mathit{s}}_t$ and would require two steps to determine. Suppose there are five possible discrete actions, $\pi \left({\mathit{s}}_t\right)$ will first decide on the most probable action and use it as its decision. Nevertheless, due to the greedy selection, the final choice may not be the same as the action with the highest possibility.

In this paper, the action consists of six joint torques; there is an infinitely possible choice of actions because the torque is a continuous variable. As a result, the possibility of all activities being covered cannot be covered. Instead of using $\pi \left({\mathit{s}}_t\right)$ to determine the most probable action, the algorithm resolves to using $\mathit{T}\left({\mathit{s}}_t\right)$ to directly determine the action $a_i$, which consists of the six joint torques, based on the states ${\mathit{s}}_t$. The algorithm also added noise to the action to ensure value function compatibility.

The modified DDPG method (DDPG-1) algorithm is shown as Algorithm 2.

Algorithm 2: DDPG-1

• Initialize $\mathit{\theta }$ and $\mathit{\varphi }$ arbitrarily, set $\widehat{\mathit{\theta }}=\mathit{\theta }$ and $\widehat{\mathit{\varphi }}=\mathit{\varphi }$, iterate until converge;
• For every episode.
  • For every time step $t$:
    • Given a state ${\mathit{s}}_t$, take action ${\mathit{a}}_t$ based on $\mathit{T}$;
    • Obtain reward $r_t$ and each new state ${\mathit{s}}_{t+1}$;
    • Store all (${\mathit{s}}_t$,${\mathit{a}}_t$, $r_t$, ${\mathit{s}}_{t+1}$) into the buffer.
  •  
    • Sample a batch of (${\mathit{s}}_i$, ${\mathit{a}}_i$, $r_i$, ${\mathit{s}}_{i+1}$) from the buffer;
    • Set the cost function.

$$C\left(\mathit{\theta }\right)={\displaystyle \sum }_i{\left[r_i+\gamma \widehat{Q}\left({\mathit{s}}_{i+1},\widehat{\mathit{T}}\left({\mathit{s}}_{i+1}\right)\right)-Q\left({\mathit{s}}_i,\mathit{T}\left({\mathit{s}}_i\right)\right)\right]}^2$$

(5)

Update the parameters of $Q$, which is $\mathit{\theta }$ to minimize $C\left(\mathit{\theta }\right)$.

$${\mathit{\theta }}^{*}=\mathit{\theta }-\eta \frac{\partial C\left(\mathit{\theta }\right)}{\partial \mathit{\theta }}$$

(6)

Set the value function:

$$V\left(\mathit{\varphi }\right)={\displaystyle \sum }_{i}Q\left({\mathit{s}}_i,\mathit{T}\left({\mathit{s}}_i\right)\right)$$

(7)

Update the parameters of $\mathit{T}$, which is $\mathit{\varphi }$ to maximize $V\left(\mathit{\varphi }\right)$.

$${\mathit{\varphi }}^{*}=\mathit{\varphi }+\lambda \frac{\partial V\left(\mathit{\varphi }\right)}{\partial \mathit{\varphi }}=\mathit{\varphi }+\lambda {\displaystyle \sum }_i\frac{\partial Q\left({\mathit{s}}_i,\mathit{T}\left({\mathit{s}}_i\right)\right)}{\partial \mathit{\varphi }}=\mathit{\varphi }+\lambda {\displaystyle \sum }_i\frac{\partial Q\left({\mathit{s}}_i,\mathit{T}\left({\mathit{s}}_i\right)\right)}{\partial \mathit{\varphi }}=\mathit{\varphi }+\lambda {\displaystyle \sum }_i\frac{\partial Q\left({\mathit{s}}_i,\mathit{T}\left({\mathit{s}}_i\right)\right)}{\partial \mathit{T}\left({\mathit{s}}_i\right)}\frac{\partial \mathit{T}\left({\mathit{s}}_i\right)}{\partial \mathit{\varphi }}$$

(8)

• Update $\widehat{\mathit{\theta }}=\tau {\mathit{\theta }}^{*}+\left(1-\tau \right)\mathit{\theta }$ and $\widehat{\mathit{\varphi }}=\tau {\mathit{\varphi }}^{*}+\left(1-\tau \right)\mathit{\varphi }$, where $\tau$ is a factor between 0 and 1.

Using the DDPG-1 method, one can consistently update the parameters of the critic $Q$ and the policy $\mathit{T}$ consistently during training to obtain the optimized critic $Q$ and policy $\mathit{T}$. For each training step, the action selected is ${\mathit{a}}_t=\mathit{T}\left(V;\varphi \right)+\mathit{N}$, where $\mathit{N}$ is the noise for value function compatibility. We also explored the TD3 algorithm, but it did not provide too much improvement.

2.3. The Differential Kinematics

The robotic control is slightly more complicated than the standard control systems. Unlike the regular control system that directly takes the control error for feedback, the control system needs to adequately translate the trajectory error in the task space into the various joint torques. In other words, there is a transformation of the measurement from the “task space” into the “joint space,” and the translation involves the differentiation of the Jacobian matrices, which, in turn, introduces Coriolis accelerations. The robotic control system will need to cancel or suppress this nonlinear effect. In a common end-following task, the position and attitude of the robot end-effector in the Cartesian coordinate system contain three position values ($P_x$, $P_y$, $\mathrm{and} P_z$) and three attitude angles ($\alpha$,$\beta$,$\mathrm{and} \gamma$), for a total of six degrees of freedom. For an RA 605 robotic arm with six joints, the controller will need to perform the inverse kinematics and compute the nonlinear dynamic forces for cancellation and exercise the control. One of the goals of this research is to use a deep learning network for the inverse dynamic model and directly control the robotic arm.

3. The Reinforcement Learning Structure

The RL Network Structure

The following section will briefly describe the deep learning network setup. For the agent, we set 43 observed results as the inputs, including the roll angle $\varphi$, pitch angle $\theta$, yaw angle $\psi$, the roll angular velocity $\dot{\varphi }$, pitch angular velocity $\dot{\theta }$, yaw angular velocity $\dot{\psi }$, the roll angular acceleration $\ddot{\varphi }$, pitch angular acceleration $\ddot{\theta }$, and yaw angular acceleration $\ddot{\psi }$. The measurements were made with respect to the end of the last joint relative to the reference coordinate. In addition, there are also three positions ($P_x,P_y,\mathrm{and} P_z$), velocities ($V_x,V_y,\mathrm{and} V_z$), and accelerations ($A_x,A_y,\mathrm{and} A_z$) of the endpoint of the robotic arm, also based on the reference coordinate. Furthermore, there are six joint angles ($q_1~q_6$), six joints angular velocities (${\dot{q}}_1~{\dot{q}}_6$), six joints angular accelerations (${\ddot{q}}_1~{\ddot{q}}_6$), six joints torques (${\tau }_1~{\tau }_6$), and the total power of joints ($P_{joint\mathit{s}}$). The outputs of the actor network are the six joint torques.

Although deep learning is a powerful tool, the computation effort involved in training the network can be too great for practical applications. Our initial attempt to help with the training was to separate the effect of the positioning error and the effect of the velocity error to prevent the conflicting demands from holding position and moving the arm. However, the learning behavior of the network was not predictable. After many attempts, we finally came up with a unique structure for the actor network. The actor network consists of two parallel networks, each with three 225-neuron fully connected hidden layers and a fully connected output layer, as shown in Figure 1.

In addition, we also break down the critic network into two parts: one for observation and the other for the robot. The observation part of the network takes the 43 observed results as inputs, and the robot part takes the six joint torques as inputs. The two parts both consist of three fully connected hidden layers with 225 neurons. Their outputs combine to form the network Q value output, as shown in Figure 2.

It is now easy to use the RL blocks for training, as shown in Figure 3, consisting of the Observation, Reward, Agent, Robot, and Sensor block. The Observation block receives the resultant states reflecting the attitude of the end effector with 43 inputs, as shown in Figure 4. The Reward block diagram receives the result states and calculates the reward, as shown in Figure 5. The Agent gets the output from the Observation, Reward, and Check-If-Done block, updates the Q function and the $\pi$ function, and outputs the reaction torques to the robot. The Sensor computes the forward kinematics as the resulting states and feeds it back to the three blocks.

4. Model Predictive Control [11,12]

As described before, MPC is based on the system dynamic model. The robot is described by Equation (9). Define the control law, Equation (10), to cancel the nonlinear effect:

$$\mathit{M}\left(\mathit{\theta }\right)\ddot{\mathit{\theta }}+\mathit{C}\left(\mathit{\theta },\dot{\mathit{\theta }}\right)+\mathit{G}\left(\mathit{\theta }\right)=\mathit{\tau }$$

(9)

$$\mathit{\tau }=\mathit{M}\left(\mathit{\theta }\right)\mathit{v}+\mathit{C}\left(\mathit{\theta },\dot{\mathit{\theta }}\right)+\mathit{G}\left(\mathit{\theta }\right)$$

(10)

The system reduces to a feedback-linearized system (11).

$$\ddot{\mathit{\theta }}=\mathit{v}=\mathrm{cons}t$$

(11)

In the MPC, we apply a proportional-derivative (PD) controller given by Equation (12) to stabilize the robotic arm system:

$$\mathit{v}=k_1\left({\mathit{\theta }}_d-\mathit{\theta }\right)+k_2\left({\dot{\mathit{\theta }}}_d-\dot{\mathit{\theta }}\right)$$

(12)

where ${\mathit{\theta }}_d$ is the vector of joint variables to reflect the desired trajectory. Consider a time interval $\left[t,t+h\right]$, where $h$ is the timeslot of the prediction. From Equation (10), we can develop the following two prediction models [12]:

$$\dot{\mathit{\theta }}\left(t+h\right)=\mathit{v}h+\dot{\mathit{\theta }}\left(t\right)$$

(13)

$$\mathit{\theta }\left(t+h\right)=\frac{1}{2}\mathit{v}{h}^2+\dot{\mathit{\theta }}\left(t\right)h+\mathit{\theta }\left(t\right)$$

(14)

If the target joint angle is ${\mathit{\theta }}_d$, a simplified MPC design is to define a one-horizon time quadratic cost function to stabilize the system as:

$$J={\mathit{e}}^2\left(t+h\right)+\rho {\dot{\mathit{e}}}^2\left(t+h\right)$$

(15)

where $\mathit{e}\left(t+h\right)={\mathit{\theta }}_d-\mathit{\theta }\left(t+h\right)$ is the predicted angular position error, and $\dot{\mathit{e}}\left(t+h\right)=0-\dot{\mathit{\theta }}\left(t+h\right)$ is the predicted angular velocity error. In Equation (7), the timeslot $h$ and weight $\rho$ are positive real control parameters to be designed. To minimize $J$, one introduces the prediction model Equation (13) into Equation (14) and lets the partial derivative $\partial J/\partial v$ equal 0. Considering that the robot slowly follows the trajectory, the resulting control becomes:

$$\mathit{v}=k_1\left({\mathit{\theta }}_d-\mathit{\theta }\left(t\right)\right)-k_2\dot{\mathit{\theta }}\left(t\right)$$

(16)

where $k_1$ and $k_2$ are:

$$k_1=\frac{2}{h^2+4\rho } \mathrm{and} k_2=\frac{2h^2+4\rho }{h^3+4\rho h}$$

(17)

By allowing the target joint angle to change over time, and from Equation (8), we obtain a more general control law:

$$\mathit{v}\left(t\right)=k_1\left({\mathit{\theta }}_d\left(t\right)-\mathit{\theta }\left(t\right)\right)-k_2\dot{\mathit{\theta }}\left(t\right)$$

(18)

From Equations (9), (10), and (18), a block diagram of the MPC controller and the robotic arm system can be shown in Figure 6, where ${\mathit{\theta }}_d\left(t\right)$ are the target joint angles in the joint-space.

The main goal of the control is for the joint angles to follow their targets according to the second-order dynamics specified by the transfer function ${\omega }_0^2/({\mathit{s}}^2+2\zeta {\omega }_0+{\omega }_0^2)$, where $\zeta$ is a damping factor and ${\omega }_0$ is a natural frequency by design.

Modeling of the Robotic Arm System

The HIWIN Technologies Corporation provided the RA605 robot arm for the D-H table, which can be readily incorporated into the “rvctools” Robotics toolbox in MATLAB for simulation purposes. It is also possible to obtain the system parameters for the dynamic model of the robot.

This research applied the DDPG to the RL training algorithm. We built an RA605 robotic arm model within the MATLAB simulation environment and, for generality purposes, four different target trajectories: a small circle trajectory (0.1 m diameter), a larger circle trajectory (0.2 m diameter), a small square trajectory (0.1 m side length) and a big square trajectory (0.2 m side length), Figure 7. To demonstrate the ability of six-axis trajectory control, we fixed the roll, pitch, and yaw angles of the end effector at $\pi$, $-\pi /2$, and 0 rad, respectively.

5. The Control Results

This section compares the control performance of the RL and MPC trajectory controls.

5.1. The RL Training Results

The 0.1 m diameter circle trajectory took about 343,070 s to train on an i7-computer running GPU and had 22,228 episodes. Figure 8 shows the roll, pitch, and yaw angle tracking results in the task space.

The 0.2 m diameter circle trajectory took a shorter 65,783 s and 4378 episodes to train. The roll, pitch, and yaw angle tracking results in the task space are shown in Figure 9.

The 0.1 m square trajectory converged after training for about 186,100 s and 12,284 episodes. In Figure 10, we show the roll, pitch, and yaw angle tracking results in the task space.

The 0.2 m square trajectory took about 129,040 s and 8338 episodes to train. In Figure 11, we show the roll, pitch, and yaw angle tracking results in the task space.

5.2. The Twin Delayed Deep Deterministic Policy Gradient Training

The authors also constructed a Twin Delayed Deep Deterministic Policy Gradient (TD3) network for comparison purposes. The network took 25,000 episodes to reach stable control. Figure 12 shows the resulting response. The robot arm’s trajectory follows a short 7.5 mm vertical line. As one observes from the result, Figure 12, the robot arm moved too fast and oscillated around a position with a 30 mm offset.

5.3. The MPC Results

For the MPC, we selected $h=0.003 \mathrm{s}$ and $\rho =1.04\times {10}^{-5}$ to achieve a closed-loop bandwidth of ${\omega }_n=200 \mathrm{rad}/\sec$ and $\zeta =1$ for no oscillation. From Equation (17), $k_1=4\times {10}^4$, $k_2=400$.

The block diagram for the control system is shown in Figure 13. The MPC controller consists of the inverse dynamic calculation and uses the values of $v$, $q$, and $\dot{q}$ for the input and the joint torques $\tau$ for the output. The parameter $v$ is the constant angular acceleration produced by the MPC controller. Four different target trajectories are shown in Figure 14.

For comparison, we ran the same trajectories as in the previous tests. The roll, pitch, and yaw angles in the task space are set at $\pi$,$-\pi /2$, and 0 rad, respectively. The motion is constrained to a slow speed.

To compare with the RL method easily, we put the result of MPC together with the RL method. For the 0.1 m diameter circle trajectory, we show the roll, pitch, and yaw angle tracking results in the task space in Figure 8. Figure 9 shows 0.2 m diameter circle roll, pitch, and yaw angle tracking results in the task space. The results for the 0.1 m square trajectory are shown in Figure 10. Again, Figure 10 shows the roll-pitch-yaw trajectories. Finally, the results for the 0.2 m square target trajectory are shown in Figure 11. The roll-pitch-yaw angle tracking results are shown in Figure 11.

5.4. The Comparison of RL and MPC

Table 1. The roll angle of the end of the robotic arm (0.1 m circle).

Time (s)	Roll_MPC (Rad)	Roll_RL (Rad)	Roll_Demand (Rad)	Error_MPC (Rad)	Error_RL (Rad)
0	3.14	3.14	3.14	4.07 × 10−4	1.43 × 10−4
0.5	3.14	3.14	3.14	3.87 × 10−4	−8.90 × 10−4
1	3.14	3.14	3.14	4.33 × 10−4	−4.98 × 10−3
1.5	3.14	3.13	3.14	3.96 × 10−4	−8.74 × 10−3
2	3.14	3.14	3.14	3.61 × 10−4	−5.75 × 10−3
2.5	3.14	3.15	3.14	4.83 × 10−4	4.35 × 10−3
3	3.14	3.14	3.14	4.21 × 10−4	6.84 × 10−4
3.5	3.14	3.14	3.14	3.29 × 10−4	−1.58 × 10−4
4	3.14	3.14	3.14	4.37 × 10−4	−4.29 × 10−4
4.5	3.14	3.14	3.14	4.88 × 10−4	1.37 × 10−3

,

Table 2. The pitch angle of the end of the robotic arm (0.1 m circle).

Time (s)	Pitch_MPC (Rad)	Pitch_RL (Rad)	Pitch_Demand (Rad)	Error_MPC (Rad)	Error_RL (Rad)
0	−1.57	−1.57	−1.57	6.66 × 10−15	−4.07 × 10−4
0.5	−1.57	−1.57	−1.57	7.67 × 10−4	−7.58 × 10−4
1	−1.57	−1.57	−1.57	7.87 × 10−4	−2.18 × 10−3
1.5	−1.57	−1.57	−1.57	8.05 × 10−4	−1.74 × 10−3
2	−1.57	−1.57	−1.57	8.11 × 10−4	−3.28 × 10−3
2.5	−1.57	−1.57	−1.57	8.05 × 10−4	1.45 × 10−3
3	−1.57	−1.57	−1.57	7.89 × 10−4	−1.96 × 10−4
3.5	−1.57	−1.58	−1.57	7.69 × 10−4	−4.39 × 10−3
4	−1.57	−1.56	−1.57	7.23 × 10−4	7.04 × 10−3
4.5	−1.57	−1.57	−1.57	7.96 × 10−4	6.37 × 10−4

and

Table 3. The yaw angle of the end of the robotic arm (0.1 m circle).

Time (s)	Yaw_MPC (Rad)	Yaw_RL (Rad)	Yaw_Demand (Rad)	Error_MPC (Rad)	Error_RL (Rad)
0	7.36 × 10−15	−2.05 × 10−4	0	7.36 × 10−15	−2.05 × 10−4
0.5	−9.65 × 10−5	8.12 × 10−3	0	−9.65 × 10−5	8.12 × 10−3
1	9.61 × 10−5	2.68 × 10−3	0	9.61 × 10−5	2.68 × 10−3
1.5	−3.96 × 10−5	9.70 × 10−4	0	−3.96 × 10−5	9.70 × 10−4
2	−5.90 × 10−5	3.90 × 10−3	0	−5.90 × 10−5	3.90 × 10−3
2.5	2.90 × 10−5	7.66 × 10−4	0	2.90 × 10−5	7.66 × 10−4
3	3.38 × 10−5	−8.37 × 10−3	0	3.38 × 10−5	−8.37 × 10−3
3.5	−8.86 × 10−5	−7.24 × 10−3	0	−8.8 × 10−5	−7.24 × 10−3
4	9.58 × 10−5	4.31 × 10−3	0	9.58 × 10−5	4.31 × 10−3
4.5	5.85 × 10−5	−3.24 × 10−5	0	5.85 × 10−5	−3.24 × 10−5

show the comparison for the roll-pitch-yaw angle tracking effect for the 0.1 m circular trajectory.

Table 4. The roll angle of the end of the robotic arm (0.2 m circle).

Time (s)	Roll_MPC (Rad)	Roll_RL (Rad)	Roll_Demand (Rad)	Error_MPC (Rad)	Error_RL (Rad)
0	3.14	3.14	3.14	4.07 × 10−4	1.43 × 10−4
0.5	3.14	3.15	3.14	4.72 × 10−4	5.14 × 10−3
1	3.14	3.14	3.14	4.40 × 10−4	−1.71 × 10−3
1.5	3.14	3.14	3.14	3.40 × 10−4	−3.64 × 10−3
2	3.14	3.14	3.14	4.28 × 10−4	−3.75 × 10−3
2.5	3.14	3.14	3.14	3.46 × 10−4	−5.92 × 10−3
3	3.14	3.13	3.14	4.54 × 10−4	−1.10 × 10−2
3.5	3.14	3.15	3.14	3.25 × 10−4	8.85 × 10−3
4	3.14	3.14	3.14	4.59 × 10−4	−6.69 × 10−4
4.5	3.14	3.15	3.14	4.75 × 10−4	6.82 × 10−3

,

Table 5. The pitch angle of the end of the robotic arm (0.2 m circle).

Time (s)	Pitch_MPC (Rad)	Pitch_RL (Rad)	Pitch_Demand (Rad)	Error_MPC (Rad)	Error_RL (Rad)
0	−1.57	−1.57	−1.57	6.66 × 10−15	−4.07 × 10−4
0.5	−1.57	−1.60	−1.57	6.66 × 10−4	−2.44 × 10−2
1	−1.57	−1.58	−1.57	6.86 × 10−4	−8.60 × 10−3
1.5	−1.57	−1.57	−1.57	8.67 × 10−4	−2.43 × 10−3
2	−1.57	−1.57	−1.57	9.46 × 10−4	−1.66 × 10−3
2.5	−1.57	−1.57	−1.57	8.12 × 10−4	−1.51 × 10−3
3	−1.57	−1.58	−1.57	7.75 × 10−4	−1.00 × 10−2
3.5	−1.57	−1.60	−1.57	6.23 × 10−4	−2.93 × 10−2
4	−1.57	−1.55	−1.57	6.47 × 10−4	2.46 × 10−2
4.5	−1.57	−1.57	−1.57	7.96 × 10−4	−1.13 × 10−3

and

Table 6. The yaw angle of the end of the robotic arm (0.2 m circle).

Time (s)	Yaw_MPC (Rad)	Yaw_RL (Rad)	Yaw_Demand (Rad)	Error_MPC (Rad)	Error_RL (Rad)
0	−3.64 × 10−14	−2.05 × 10−4	0	−3.64 × 10−14	−2.05 × 10−4
0.5	6.45 × 10−5	1.66 × 10−2	0	6.45 × 10−5	1.66 × 10−2
1	3.30 × 10−5	1.90 × 10−3	0	3.30 × 10−5	1.90 × 10−3
1.5	−6.78 × 10−5−5	−5.11 × 10−3	0	−6.78 × 10−5	−5.11 × 10−3
2	2.04 × 10−5	1.51 × 10−2	0	2.04 × 10−5	1.51 × 10−2
2.5	−6.10 × 10−5	−7.64 × 10−3	0	−6.10 × 10−5	−7.64 × 10−3
3	4.69 × 10−5	1.97 × 10−3	0	4.69 × 10−5	1.97 × 10−3
3.5	−8.21 × 10−5	1.38 × 10−2	0	−8.21 × 10−5	1.38 × 10−2
4	5.19 × 10−5	−6.25 × 10−3	0	5.19 × 10−5	−6.25 × 10−3
4.5	6.77 × 10−5	−3.48 × 10−2	0	6.77 × 10−5	−3.48 × 10−2

show the comparison for the 0.2 m diameter circular trajectory roll-pitch-yaw errors. For the 0.1 m square target trajectory,

Table 7. The roll angle of the end of the robotic arm (0.1 m side length square).

Time (s)	Roll_MPC (Rad)	Roll_RL (Rad)	Roll_Demand (Rad)	Error_MPC (Rad)	Error_RL (Rad)
0	3.14	3.14	3.14	4.07 × 10−4	1.43 × 10−4
0.5	3.14	3.14	3.14	1.35 × 10−3	3.06 × 10−3
1	3.14	3.14	3.14	1.22 × 10−4	6.64 × 10−4
1.5	3.14	3.14	3.14	1.05 × 10−3	1.46 × 10−3
2	3.14	3.14	3.14	−4.21 × 10−4	1.61 × 10−3
2.5	3.14	3.13	3.14	1.14 × 10−3	−8.94 × 10−3
3	3.14	3.14	3.14	9.41 × 10−4	−4.83 × 10−3
3.5	3.14	3.14	3.14	3.87 × 10−4	1.86 × 10−3
4	3.14	3.16	3.14	3.71 × 10−4	1.89 × 10−2
4.5	3.14	3.15	3.14	6.34 × 10−4	1.13 × 10−2

,

Table 8. The pitch angle of the end of the robotic arm (0.1 m side length square).

Time (s)	Pitch_MPC (Rad)	Pitch_RL (Rad)	Pitch_Demand (Rad)	Error_MPC (Rad)	Error_RL (Rad)
0	−1.57	−1.57	−1.57	6.66 × 10−15	−4.07 × 10−4
0.5	−1.57	−1.58	−1.57	7.73 × 10−4	−6.96 × 10−3
1	−1.57	−1.57	−1.57	6.29 × 10−4	1.10 × 10−5
1.5	−1.57	−1.57	−1.57	7.50 × 10−4	−3.08 × 10−3
2	−1.57	−1.57	−1.57	1.00 × 10−3	5.17 × 10−3
2.5	−1.57	−1.56	−1.57	1.02 × 10−3	1.08 × 10−2
3	−1.57	−1.57	−1.57	7.11 × 10−4	1.21 × 10−3
3.5	−1.57	−1.59	−1.57	6.24 × 10−4	−1.67 × 10−2
4	−1.57	−1.57	−1.57	7.10 × 10−4	5.29 × 10−3
4.5	−1.57	−1.57	−1.57	7.82 × 10−4	3.52 × 10−4

and

Table 9. The yaw angle of the end of the robotic arm (0.1 m side length square).

Time (s)	Yaw_MPC (Rad)	Yaw_RL (Rad)	Yaw_Demand (Rad)	Error_MPC (Rad)	Error_RL (Rad)
0	2.59 × 10−15	−2.05 × 10−4	0	2.59 × 10−15	−2.05 × 10−4
0.5	9.02 × 10−4	9.79 × 10−3	0	9.02 × 10−4	9.79 × 10−3
1	−7.33 × 10−4	2.65 × 10−3	0	−7.33 × 10−4	2.65 × 10−3
1.5	7.69 × 10−4	−1.11 × 10−2	0	7.69 × 10−4	−1.11 × 10−2
2	−5.98 × 10−4	−1.50 × 10−2	0	−5.98 × 10−4	−1.50 × 10−2
2.5	5.06 × 10−4	−1.83 × 10−2	0	5.06 × 10−4	−1.83 × 10−2
3	6.57 × 10−4	−1.55 × 10−2	0	6.57 × 10−4	−1.55 × 10−2
3.5	−1.76 × 10−4	−7.55 × 10−3	0	−1.76 × 10−4	−7.55 × 10−3
4	−5.32 × 10−4	−8.68 × 10−3	0	−5.32 × 10−4	−8.68 × 10−3
4.5	5.47 × 10−4	−5.36 × 10−3	0	5.47 × 10−4	−5.36 × 10−3

show the comparison of the attitude errors. Likewise,

Table 10. The roll angle of the end of the robotic arm (0.2 m side length square).

Time (s)	Roll_MPC (Rad)	Roll_RL (Rad)	Roll_Demand (Rad)	Error_MPC (Rad)	Error_RL (Rad)
0	3.14	3.14	3.14	−2.49 × 10−14	1.43 × 10−4
0.5	3.12	3.15	3.14	−2.21 × 10−2	4.45 × 10−3
1	3.14	3.14	3.14	4.42 × 10−4	−6.59 × 10−4
1.5	3.16	3.14	3.14	1.47 × 10−2	1.31 × 10−3
2	3.15	3.15	3.14	1.08 × 10−2	5.18 × 10−3
2.5	3.16	3.14	3.14	1.62 × 10−2	1.17 × 10−3
3	3.14	3.15	3.14	−2.24 × 10−3	9.98 × 10−3
3.5	3.12	3.16	3.14	−1.67 × 10−2	1.78 × 10−2
4	3.12	3.16	3.14	−1.88 × 10−2	1.35 × 10−2
4.5	3.13	3.16	3.14	−8.64 × 10−3	1.46 × 10−2

,

Table 11. The pitch angle of the end of the robotic arm (0.2 m side length square).

Time (s)	Pitch_MPC (Rad)	Pitch_RL (Rad)	Pitch_Demand (Rad)	Error_MPC (Rad)	Error_RL (Rad)
0	−1.57	−1.57	−1.57	−3.55 × 10−15	−4.07 × 10−4
0.5	−1.56	−1.58	−1.57	6.06 × 10−3	−1.12 × 10−2
1	−1.56	−1.57	−1.57	1.05 × 10−2	−2.60 × 10−3
1.5	−1.55	−1.55	−1.57	1.63 × 10−2	2.25 × 10−2
2	−1.54	−1.55	−1.57	2.91 × 10−2	1.81 × 10−2
2.5	−1.53	−1.57	−1.57	4.42 × 10−2	5.35 × 10−3
3	−1.52	−1.58	−1.57	5.00 × 10−2	−5.54 × 10−3
3.5	−1.52	−1.58	−1.57	5.12 × 10−2	−1.39 × 10−2
4	−1.53	−1.57	−1.57	4.13 × 10−2	−1.34 × 10−3
4.5	−1.53	−1.57	−1.57	4.00 × 10−2	−1.21 × 10−3

and

Table 12. The yaw angle of the end of the robotic arm (0.2 m side length square).

Time (s)	Yaw_MPC (Rad)	Yaw_RL (Rad)	Yaw_Demand (Rad)	Error_MPC (Rad)	Error_RL (Rad)
0	2.98 × 10−14	−2.05 × 10−4	0	2.98 × 10−14	−2.05 × 10−4
0.5	2.93 × 10−2	−9.36 × 10−3	0	2.93 × 10−2	−9.36 × 10−3
1	2.36 × 10−2	−1.89 × 10−3	0	2.36 × 10−2	−1.89 × 10−3
1.5	2.23 × 10−2	1.56 × 10−2	0	2.23 × 10−2	1.56 × 10−2
2	2.62 × 10−2	−7.73 × 10−3	0	2.62 × 10−2	−7.73 × 10−3
2.5	2.10 × 10−2	−1.29 × 10−2	0	2.10 × 10−2	−1.29 × 10−2
3	2.98 × 10−2	5.38 × 10−3	0	2.98 × 10−2	5.38 × 10−3
3.5	2.55 × 10−2	4.18 × 10−3	0	2.55 × 10−2	4.18 × 10−3
4	2.88 × 10−2	1.48 × 10−2	0	2.88 × 10−2	1.48 × 10−2
4.5	2.35 × 10−2	1.45 × 10−3	0	2.35 × 10−2	1.45 × 10−3

compare the tracking results of the 0.2 m square trajectory.

From Table 1, Table 2, Table 3, Table 4, Table 5, Table 6, Table 7, Table 8 and Table 9, most (88%) of the absolute values of the roll-pitch-yaw angular errors resulting from the MPC are smaller than the corresponding errors resulting from the RL control. That means the MPC can track the designed trajectory better than the RL method. However, in Table 10, Table 11 and Table 12, most (83%) of the roll-pitch-yaw show larger errors by MPC, indicating that MPC is not as effective in controlling end effector attitude.

Furthermore, looking at this experimental result, one finds that the MPC drives joints 4 and 6 into very strenuous movements and does not restore to the original position, as shown in Figure 15. After examining the trajectory, one sees that the target trajectory passes through the vicinity of the singularity. The model-based control cannot avoid violent action without further treatment. The RL control, on the other hand, instinctively avoided these movements.

When the two methods are compared, it is clear that while the MPC is very effective in dealing with highly nonlinear system dynamics, it still necessitates the use of a model and additional treatment for singularities. On the other hand, the RL requires no prior knowledge of the underlying model and can drive the robotic arm through any trajectory. In addition, it is not affected by singularity. Furthermore, as the trajectory becomes larger and more complicated, the MPC no longer shows an advantage over the RL. This may be because the MPC cannot accurately calculate the attitude using the forward kinematics while the robotic six-joint angles become too large.

6. Conclusions

This paper proposes a split network structure for the direct application of RL to robotic trajectory following control. We first show that it is possible to train the RL network to directly mimic the inverse dynamic cancellation action without the help of a precise system model. The RL uses the modified DDPG algorithm as the underlying learning algorithm, and the tests follow four different target trajectories. The 0.1 m circle trajectory spent 343,070 s and 22,228 episodes training. It took 65,783 s and 4378 episodes to learn the 0.2 m circle trajectory. The 0.1 m square trajectory took 186,100 s to train, while the 0.2 m square trajectories took 129,040 s.

This study also compares RL performance to MPC performance. The RL control inherently avoids exercising violent movements while passing through the singularities and still succeeds in handling large and complicated trajectory situations. The MPC requires an accurate system model and is straightforward in implementation. However, it cannot avoid singularity without additional treatment and fails to control in large and complicated trajectory situations.

Comparing the two methods shows that RL successfully makes the robotic arm go through the target trajectory without excessive joint angle movements. MPC does not need time for training and achieves smoother control, but it requires an accurate system model. The authors will also try to implement the controller in an experiment for future verification.