Adaptive Position Control of Pneumatic Continuum Manipulator Based on MAML Meta-Reinforcement Learning

Abstract

Reinforcement learning algorithms usually focus on a specific task, which often performs well only in the training environment. When the task changes, its performance drops significantly, with the algorithm lacking the ability to adapt to new environments and tasks. For the position control of a pneumatic continuum manipulator (PCM), there is a high degree of similarity between tasks, and the training speed of new tasks can be accelerated by utilizing the training experience from other tasks. To increase the adaptability of control policies to new tasks, this paper proposes an adaptive position control algorithm of the PCM based on the Model-Agnostic Meta-Learning (MAML) meta-reinforcement learning algorithm. The MAML meta-reinforcement learning algorithm is used to train the control strategy of PCM, and the information and experience collected during the training process across multiple tasks are used to quickly learn the control policy for new tasks, improving the ability of PCM to quickly adapt to new tasks. The experimental results demonstrate that after training with the MAML meta-reinforcement learning algorithm, the PCM can significantly reduce the training time when faced with new tasks and obtain the control policies suitable for these new tasks.

1. Introduction

The continuum manipulator is a bionic, slender super-redundant manipulator that provides superior reach and dexterity capabilities in cluttered environments. However, because of its high nonlinearity, the precise control of a continuum manipulator still remains a significant challenge.

Reinforcement learning is a model-free control method and has been used for the control of continuum manipulators. You [1] and Sreeshankar [2] used the Q-Learning algorithm and DQN algorithm to train the control policy for a planar continuum manipulator, achieving the position control of the manipulator. Sreeshankar [3] and Li [4] applied the Deep Deterministic Policy Gradient (DDPG) algorithm to the control of a continuum manipulator, achieving end-effector path tracking control. Centurelli [5] used the Trust Region Policy Optimization (TRPO) algorithm to learn the controller policy of the continuum manipulator, achieving position control of the continuum manipulator.

Although reinforcement learning algorithms have achieved great success in PCM control, they ignore the similarities between tasks, making them unable to effectively utilize training experiences from other tasks. Consequently, when faced with new tasks, the learning process often has to start from scratch, consuming substantial time and resources [6]. Therefore, how to use previous experience to improve the adaptability of agents to new tasks has become an important research direction. In order to enable the agent to have the ability to learn and adapt to new tasks quickly, researchers have proposed various solutions. Policy Distillation [7] extracts experience from multiple policies to form a more generalized comprehensive policy. This method can improve the adaptability of the strategy through self-distillation or cross-task distillation. Multi-task learning [8] improves the generalization ability of an agent by simultaneously learning on multiple related tasks. However, the success of the above methods usually depends on the correlation between tasks. If the differences between tasks are too great, the effect of experience sharing may not be obvious, and it may even interfere with the learning process.

Some scholars have proposed the concept of meta-reinforcement learning, combining meta-learning algorithms with reinforcement learning algorithms based on the ability of meta-learning to rapidly learn with only a small number of samples [9]. Meta-reinforcement learning trains on multiple tasks, allowing agents to quickly adapt to new tasks with a small number of learning samples. Unlike traditional reinforcement learning, which optimizes control strategies for a single task, meta-reinforcement learning uses multiple tasks as its training samples. By training on multiple tasks, the agent acquires the ability to learn quickly. When faced with a new task, it is possible to rapidly learn the policy that adapts to the new task with only a small number of training samples by leveraging prior knowledge.

Typical meta-reinforcement learning algorithms include the memory context-based $R{L}^2$ algorithm [10], the SNAIL algorithm [11], and the CMRL algorithm [12]. These methods are easily combined with other learning algorithms and have a variety of network structures. However, complex learning tasks may require more complex models to process data and tasks, as well as more computational resources and training time. There are also task-inference-based algorithms, such as PEARL [13], MetaCURE [14], and MQL [15]. However, such methods may ignore some specific information related to the task during the task inference process, and their reliance on specific environmental conditions makes them difficult to generalize across different tasks [16]. Additionally, there are gradient descent-based algorithms, such as MAML meta-reinforcement learning [17]. As a novel meta-learning framework, MAML has spawned numerous improvements and extensions, including FO-MAML, ES-MAML [18], Reptile [19], MAESN [20], and Taming MAML [21]. The advantage of these algorithms lies in their model-agnostic nature, allowing them to be applied in various reinforcement learning scenarios and offering good generalization performance.

At the same time, there are also some works that applied meta-reinforcement learning algorithms to the control of manipulators. Li [22] combined the MQL algorithm with DDPG to propose a sample-efficient and generalizable method for manipulator skill learning. This method uses information such as the state of the end-effector and target pose as input to learn a policy that controls the manipulator’s end-effector to reach the desired pose. Hao [23] combined the MAML meta-learning algorithm with the proximal policy optimization algorithm to design the M-PPO meta-reinforcement learning algorithm for learning the task of turning on a light with a manipulator. Schoettler [24] studied meta-reinforcement learning for industrial assembly tasks, using the PEARL algorithm to learn the latent representations of a task, enabling the rapid generation of policy for the current task based on its representation. Peng [25] integrated the Particle Swarm Optimization (PSO) algorithm with MAML, proposing the PSO-MAML algorithm. This algorithm was applied to the learning of grasping tasks, enabling the manipulator to quickly adapt to new tasks. Wang [26] proposed an offline policy meta-reinforcement learning algorithm SMRL-TO and applied it to the policy training of pushing tasks, door opening tasks, and assembly tasks of the manipulator, successfully achieving rapid adaptation of the manipulator to new tasks.

Meta-reinforcement learning has demonstrated strong generalization capabilities in the application of manipulators, but it has not yet been applied to the control of continuum manipulators. Because of the model-agnostic nature of the MAML algorithm, it can be applied in different reinforcement learning scenarios, and it exhibits good generalization performance. This paper employs the MAML meta-reinforcement learning algorithm to train the control policy of pneumatic continuum manipulators, improving the ability of pneumatic continuum manipulators to rapidly adapt to new tasks.

2. MAML Meta-Reinforcement Learning

The core objective of MAML meta-reinforcement learning is to train a set of initial policy parameters such that the policy initialized with these parameters can adapt to a new task rapidly by fine-tuning the parameters with minimal interaction dates.

The training process of MAML meta-reinforcement learning involves an inner reinforcement learning loop and an outer reinforcement learning loop.

Within the inner loop, the agent randomly samples K different training tasks ${\mathcal{T}}_i$ from the task distribution $p\left(\mathcal{T}\right)$ and interacts with the environment for each task ${\mathcal{T}}_i$ to collect data ${\mathcal{D}}_i$. Let ${\pi }_{\theta }$ represent the agent’s policy, where $\theta$ represents the policy parameters. The interaction process generates an action $a_t$ based on the observed current state $a_t$ according to the policy. The environment moves to the next state $s_t+1$ and generates a reward $r_t$ after the agent executes the action. The expected reward for each task ${\mathcal{T}}_i$ can be expressed as

$${\mathcal{L}}_{T_i}\left(f_{\theta }\right)=E_{x_t,a_t\sim f_{\varphi },{\mathcal{T}}_i}\left[\sum _{t=1}^H{r}_i(x_t,a_t)\right]$$

(1)

The policy parameters are updated by maximizing the expected reward in the inner loop, and the update formula can be expressed as

$${{\theta }_i}^{\prime }=\theta -\alpha {\nabla }_{\theta }{\mathcal{L}}_{{\mathcal{T}}_i}\left(f_{\theta }\right)$$

(2)

Within the outer loop, the parameters of the policy are updated according to the total expected rewards collected from each task. Within the inner loop, the agent interacts with the environment for each task ${\mathcal{T}}_i$ to collect data ${{\mathcal{D}}_i}^{\prime }$ using the new policy ${\pi }_{{\theta }^{\prime }}$, which was trained in the inner loop. Sum up the expected return ${\mathcal{L}}_{{{\mathcal{T}}_i}^{\prime }}\left(f_{{{\theta }_i}^{\prime }}\right)$ for each task to obtain ${\displaystyle \sum _{i=1}^K}{\mathcal{L}}_{{{\mathcal{T}}_i}^{\prime }}\left(f_{{{\theta }_i}^{\prime }}\right)$. The policy parameters are updated by maximizing the sum of the expected returns ${\displaystyle \sum _{i=1}^K}{\mathcal{L}}_{{{\mathcal{T}}_i}^{\prime }}\left(f_{{{\theta }_i}^{\prime }}\right)$, and the update formula can be expressed as

$$\theta =\theta -\beta {\nabla }_{\theta }\sum _{i=1}^K{\mathcal{L}}_{{{\mathcal{T}}_i}^{\prime }}\left(f_{{{\theta }_i}^{\prime }}\right)$$

(3)

The overall training procedure of the MAML meta-reinforcement learning algorithm is illustrated in Algorithm 1.

Algorithm 1: MAML meta-reinforcement learning

Require: Task distribution

Require: Learning rates

1: Randomly initialize policy parameters

2: while not done do:

3:    Sample a batch of tasks

4:    if all ${\mathcal{T}}_i$ do:

5:       Interact with the environment using policy $f_{\theta }$ to collect data ${\mathcal{D}}_i$

6:       Calculate ${\nabla }_{\theta }{\mathcal{L}}_{{\mathcal{T}}_i}\left(f_{\theta }\right)$ based on ${\mathcal{D}}_i$ and ${\mathcal{L}}_{{\mathcal{T}}_i}$

7:       Update ${{\theta }_i}^{\prime }=\theta -\alpha {\nabla }_{\theta }{\mathcal{L}}_{{\mathcal{T}}_i}\left(f_{\theta }\right)$

8:       Interact with the environment using policy $f_{{{\theta }^{\prime }}_i}$ to collect data ${{\mathcal{D}}_i}^{\prime }$

9:    end for

10:    Update $\theta =\theta -\beta {\nabla }_{\theta }{\displaystyle \sum _{i=1}^K}{\mathcal{L}}_{{{\mathcal{T}}_i}^{\prime }}\left(f_{{{\theta }_i}^{\prime }}\right)$

11: end while

2.1. Inner Reinforcement Learning Loop

The inner loop is updated by REINFORCE. The REINFORCE algorithm follows a straightforward idea of performing gradient ascent on the parameters of the policy $\theta$ to gradually improve the performance of the policy ${\pi }_{\theta }$. The learning objective of REINFORCE is to maximize the expected reward, and the update formula can be expressed as

$$G_t=\sum _{k=t+1}^T{\gamma }^{k-t-1}{r}_k$$

(4)

where $\gamma$ is the discount factor.

The policy parameters $\theta$ are updated by maximizing the expected reward using the gradient descent method. The update formula is as follows:

$${\theta }_{t+1}={\theta }_t-\eta G_t\ln \pi \left(a_t\right|s_t,{\theta }_t)$$

(5)

But, REINFORCE has been observed to suffer a large variance when estimating the gradient. To alleviate the large variance problem, a baseline $b\left(s_t\right)$ can be subtracted from the expected reward $G_t$, where $b\left(s_t\right)$ is a function only depending on $s_t$. One approach to selecting the baseline is to use the expected return starting from state $s_t$ under the current policy as the baseline; that is, $b\left(s_t\right)=E[r_t+r_{t+1}+\dots +r_{T-1}]$. With the baseline $b\left(s_t\right)$, the update formula of policy parameters can be represented as

$${\theta }_{t+1}={\theta }_t-\eta \left(G_t-b\left(s_t\right)\right){\nabla }_{{\theta }_t}\ln \pi \left(a_t\right|s_t,{\theta }_t)$$

(6)

2.2. Outer Reinforcement Learning Loop

The outer loop uses trust-region policy optimization (TRPO) as the optimizer. The goal of TRPO is to find an updated policy ${\pi }_{{\theta }^{\prime }}$ that improves the current policy ${\pi }_{\theta }$, that is, find better policy parameters ${\theta }^{\prime }$ based on the current parameter $\theta$ to make $J\left({\theta }^{\prime }\right)\ge J\left(\theta \right)$. $J\left(\theta \right)$ is the excepted reward from the start state under the current policy ${\pi }_{\theta }$, which can be denoted as

$$\begin{array}{cc}\hfill J\left(\theta \right)& ={\mathbb{E}}_{s_0}\left[V^{{\pi }_{\theta }}\left(s_0\right)\right]\hfill \\ \hfill & ={\mathbb{E}}_{{\pi }_{{\theta }^{\prime }}}\left[\sum _{t=0}^{\infty }{\gamma }^t{V}^{{\pi }_{\theta }}\left(s_t\right)-\sum _{t=1}^{\infty }{\gamma }^t{V}^{{\pi }_{\theta }}\left(s_t\right)\right]\hfill \\ \hfill & =-{\mathbb{E}}_{{\pi }_{{\theta }^{\prime }}}\left[\sum _{t=0}^{\infty }{\gamma }^t\left(\gamma V^{{\pi }_{\theta }}\left(s_{t+1}\right)-V^{{\pi }_{\theta }}\left(s_t\right)\right)\right]\hfill \end{array}$$

(7)

Define the advantage function $A^{{\pi }_{\theta }}(s_t,a_t)$ as

$$A^{{\pi }_{\theta }}(s_t,a_t)=r(s_t,a_t)+\gamma V^{{\pi }_{\theta }}\left(s_{t+1}\right)-V^{{\pi }_{\theta }}\left(s_t\right)$$

(8)

Based on the above equation, the difference in the excepted reward between the new and old policies can be represented as

$$\begin{array}{cc}\hfill J\left({\theta }^{\prime }\right)-J\left(\theta \right)& ={\mathbb{E}}_{s_0}\left[V^{{\pi }_{\theta \prime }}\left(s_0\right)\right]-{\mathbb{E}}_{s_0}\left[V^{{\pi }_{\theta }}\left(s_0\right)\right]\hfill \\ & ={\displaystyle \frac{1}{1-\gamma }}{\mathbb{E}}_{s\sim {\nu }^{{\pi }_{{\theta }^{\prime }}}}{\mathbb{E}}_{a\sim {\pi }_{{\theta }^{\prime }}(·|s)}\left[A^{{\pi }_{\theta }}(s,a)\right]\hfill \end{array}$$

(9)

Therefore, learning the optimal policy ${\pi }_{\theta }$ is equivalent to maximizing the bonus optimization object:

$$L_{\theta }\left({\theta }^{\prime }\right)=J\left(\theta \right)+{\mathbb{E}}_{s\sim V^{{\pi }_{\theta }^{\prime }}}{\mathbb{E}}_{a\sim {\pi }_{{\theta }^{\prime }}(·|s)}\left[A^{{\pi }_{\theta }}(s,a)\right]$$

(10)

When the old and new policies are very close, the optimization objective can be substituted with

$$\begin{array}{cc}\hfill L_{\theta }\left({\theta }^{\prime }\right)& =J\left(\theta \right)+{\mathbb{E}}_{s\sim {\nu }^{{\pi }_{\theta }}}{\mathbb{E}}_{a\sim {\pi }_{{\theta }^{\prime }}(·|s)}\left[A^{{\pi }_{\theta }}(s,a)\right]\hfill \\ \hfill & =J\left(\theta \right)+{\mathbb{E}}_{s\sim {\nu }^{{\pi }_{\theta }}}{\mathbb{E}}_{a\sim {\pi }_{\theta }(·|s)}\left[{\displaystyle \frac{{\pi }_{{\theta }^{\prime }}(a\mid s)}{{\pi }_{\theta }(a\mid s)}}{A}^{{\pi }_{\theta }}(s,a)\right]\hfill \end{array}$$

(11)

To ensure that the new and old policies are sufficiently close, TRPO uses KL divergence to measure the distance between strategies. Then, the optimization formula can be expressed as

$$\begin{array}{cc}& \underset{{\theta }^{\prime }}{max}{L}_{\theta }\left({\theta }^{\prime }\right)\hfill \\ & s.t.{\mathbb{E}}_{s\sim {\upsilon }^{{\pi }_{\theta }}}\left[D_{KL}\left({\pi }_{\theta }\left(·\mid s\right),{\pi }_{{\theta }^{\prime }}\left(·\mid s\right)\right)\right]\le \delta \hfill \end{array}$$

(12)

Performing Taylor expansions on the objective and constraint to a leading order around ${\theta }_k$, the optimization formula can be expressed as

$$\begin{array}{c}{\theta }_{k+1}=\underset{{\theta }^{\prime }}{argmax}{g}^T\left({\theta }^{\prime }-{\theta }_k\right)\hfill \\ s.t.{\displaystyle \frac{1}{2}}{\left({\theta }^{\prime }-{\theta }_k\right)}^{T}H\left({\theta }^{\prime }-{\theta }_k\right)\le \delta \hfill \end{array}$$

(13)

where g represents the gradient of the objective function with respect to the policy parameters ${\theta }^{\prime }$ and H denotes the Hessian matrix of the KL divergence between the new and old policies. The expressions for g and H are given as follows:

$$g={\nabla }_{{\theta }^{\prime }}{\mathbb{E}}_{s\sim {\upsilon }^{{\pi }_{\theta }}}{\mathbb{E}}_{a\sim {\pi }_{\theta }(·|s)}\left[{\displaystyle \frac{{\pi }_{{\theta }^{\prime }}\left(a\mid s\right)}{{\pi }_{\theta }\left(a\mid s\right)}}{A}^{{\pi }_{\theta }}\left(a\mid s\right)\right]$$

(14)

$$H=\mathrm{H}\left({\mathbb{E}}_{s\sim {\upsilon }^{{\pi }_{\theta }}}\right[D_{KL}\left({\pi }_{\theta }\right(·\left|s\right),{\pi }_{{\theta }^{\prime }}(·|s\left)\right)\left]\right)$$

(15)

This approximate problem can be analytically solved by the methods of Lagrangian duality, yielding the following solution:

$${\theta }_{k+1}={\theta }_k+\sqrt{{\displaystyle \frac{2\delta }{g^T{H}^{-1}g}}}{H}^{-1}g$$

(16)

Because the TRPO algorithm approximates the objective function and the constraint conditions with first-order and second-order approximations, respectively, during the optimization process, it does not achieve an exact solution. Consequently, the new policy may not necessarily outperform the old one or may fail to satisfy the KL divergence constraint. Therefore, TRPO performs a linear search at the end of each iteration to ensure that a new policy that meets the conditions is found. The update formula of policy parameters $\theta$ can be expressed as

$${\theta }_{k+1}={\theta }_k+{\varpi }^i\sqrt{{\displaystyle \frac{2\delta }{g^T{H}^{-1}g}}}{H}^{-1}g$$

(17)

where $\varpi \in \left(0,1\right)$ is the backtracking coefficient.

3. Position Control Policy Traning of the PCM Based on MAML Meta-Reinforcement Learning

3.1. The Structure of the PCM

The PCM consists of two sections, and each section adopts the same layout. Each section of the manipulator is composed of three contractile pneumatic muscles, which are distributed at ${120}^{\circ }$. The PCM is shown in Figure 1. The overall length of the PCM is 830 mm, with each section having a length of 415 mm and a radius of 37.5 mm. Each section of the PCM has the ability to bend in different directions. PCM is driven by pneumatic muscles by adjusting the air pressure of the pneumatic muscles to control the length changes of the pneumatic muscles. The length changes of the pneumatic muscles can cause the length of the manipulator to change and lead to local bending of the manipulator. By changing the lengths of the pneumatic muscles, the direction and radius of bending of the manipulator can be controlled to drive the manipulator to perform movements such as extension, deflection, and bending.

The task of end position control for the PCM involves randomly generating a target position within the workspace and controlling the pressure of each pneumatic muscle to drive the PCM to achieve the target position. The motion range of the PCM along the x-axis is [−200, 200] mm; along the y-axis, it is [−200, 200] mm; and, along the z-axis, it is [−100, 100] mm. The movement range of the end-effector along the x-axis is set to [−200, 200] mm; along the y-axis, it is set to [−200, 200] mm; and, along the z-axis, it is set to [−100, 100] mm. For end position control, each target position corresponds to a task, and these tasks collectively form a task distribution. The task distribution $p\left(\mathcal{T}\right)$ of the PCM is a uniform distribution within its range of motion.

3.2. Meta-Reinforcement Learning Modeling and Policy Network Structure Design

State space (s). State information is crucial for making action decisions. The state of the continuum manipulator is determined by the current pressure values of each pneumatic muscle. Therefore, the state space is composed of the current pressure values of each pneumatic muscle, denoted as $s=\left[u_{11},u_{12},u_{13},u_{21},u_{22},u_{23}\right]$ with a dimension of 6.

Action space (a). The action space encompasses the entire set of actions available to the agent. Each pneumatic muscle of the PCM can be independently pressurized and depressurized. So, the pressure increments of each pneumatic muscle $\mathsf{∆}u$ can be used as actions, that is $a=\left[\mathsf{∆}{u}_{11},\mathsf{∆}{u}_{12},\mathsf{∆}{u}_{13},\mathsf{∆}{u}_{21},\mathsf{∆}{u}_{22},\mathsf{∆}{u}_{23}\right]$, with a dimension of 6.

Reward function (r). The reward function plays a critical role in reinforcement learning training. It is primarily used to guide the agent’s behavior. For actions that align with the desired behavior, the reward function provides a positive reward, while for actions that do not, it provides a negative reward. Thus, the design of the reward function directly impacts the learning speed, operational efficiency, and the final control performance of the trained model. As the objective of this paper is to control the end-effector of the PCM to reach a specified position, the negative of the difference between the current end position and the target position can be used as the reward function to encourage the policy to control the end position of the PCM as closely as possible to the target position. So, the reward function can be expressed as

$$r=-\left[{\left(x-x_{target}\right)}^2+{\left(y-y_{target}\right)}^2+{\left(y-y_{target}\right)}^2\right]$$

(18)

where $\left(x,y,z\right)$ is end position of the PCM and $\left(x_{target},y_{target},z_{target}\right)$ is the target position.

Structure of policy network. For the task of the end position control of the PCM, the design of the policy network structure is based on the action space and state space. The policy network learns the mapping from the current state to the optimal action. In the REINFORCE algorithm, the input to the policy network is the state of the agent at the current time, and the output is the mean and variance of the normal distribution of the action taken by the agent. The policy is represented by a four-layer fully connected network, with 256 neurons in the middle two layers, and each layer uses the tanh activation function. The Adam optimizer is employed to compute the policy gradient and update the network weights.

The policy network outputs a Gaussian distribution of the actions taken by the agent. Sampling from this Gaussian distribution yields the specific action taken by the agent in the current state, along with the probability of taking that action.

4. Simulation Analysis

4.1. Simulation Environment

For simulation, the dynamic model of the PCM is used. The dynamic model of the PCM can be expressed as the mapping between the current time control input $u_t$, the previous control inputs $u_{t-1},\dots ,u_{t-T}$, the position at the current time ${\mathbf{x}}_t$, and the position at the previous time ${\mathbf{x}}_{t-1},\dots ,{\mathbf{x}}_{t-T}$. Specifically, it can be expressed as

$${\mathbf{x}}_{t+1}=f\left(u_t,u_{t-1},\dots ,u_{t-T},{\mathbf{x}}_t,{\mathbf{x}}_{t-1},\dots ,{\mathbf{x}}_{t-T}\right)$$

(19)

where T is the horizon length and f represents the dynamics model of the PCM.

This paper uses an LSTM network to model the dynamics of the PCM. The input of the LSTM network dynamic model is the pressure value of each pneumatic muscle, and the output is the end-effector position of the PCM.

Figure 2 illustrates the architecture of the proposed LSTM network for the dynamics model of the PCM. The network comprises an LSTM layer and a fully connected layer. The LSTM layer has 200 neurons, while the fully connected layer has three neurons, and the horizon length $T=3$.

The dataset is sampled using pseudorandom motor babbling, yielding 21,000 data points. Normalizing the data, 80% is used for training, with the remainder serving as a validation set. The LSTM network is trained using the Adam optimization algorithm, with the root mean square error (RMSE) as the loss function. The initial learning rate is set to 0.1, the learning rate decay factor is 0.5, the learning rate decay period is 5, the batch size is 32, and the number of training iterations is 30. The change in RMSE during training is shown in Figure 3.

As shown in Figure 3, on the training set, the RMSE converges to 0.02 after training. On the validation set, the RMSE converges to 0.03, and the model’s prediction accuracy is 91.2%. Therefore, the LSTM model is capable of representing the dynamics model of the PCM.

4.2. Training Overview of MAML Meta-Reinforcement Learning

To verify the adaptive capabilities of MAML meta-reinforcement learning in end-position control tasks for PCM, this section investigates the generalization performance of the PCM control policy trained by MAML meta-reinforcement learning through simulations.

For reinforcement learning, generalization refers to the number of steps an agent needs to take to achieve good performance in a new task. It includes two evaluation metrics, and one is the excepted reward of the agent in the new task. The excepted reward reflects the overall reward that the agent receives in new tasks. The higher the excepted reward that the agent receives in new tasks, the better its performance in new tasks. The other metric is the adaptation step number, which refers to how many steps the agent needs to interact with the environment in order to achieve a good performance in the new task. The number of adaptation steps reflects the ability of the agent to adapt to new tasks. The fewer the adaptation steps, the faster the agent adapts to the new tasks.

Different target positions are sampled from the task distribution $p\left(\mathcal{T}\right)$ as training tasks, and the agent interacts with the environment based on these different tasks. In the experiment, the agent is set to interact with the environment to generate 20 trajectories under each task, with each trajectory having a length of 10. Within the inner loop, each batch of tasks undergoes only one gradient update during the training phase. The hyperparameters of the MAML meta-reinforcement learning algorithm during the training process are set as shown in

Table 1. Hyperparameters of MAML meta-reinforcement learning.

Parameters	Values
Learning rate $\alpha$	$1\times {10}^{-14}$
Size of the trust-region $\delta$	$1\times {10}^{-2}$
Conjugate gradient iterations	10
Maximum line search iterations	15
Line search backtrack coefficient	0.8
Inner loop task samples	20
Number of trajectories collected per task	20
Inner loop optimization iterations	1
Outer loop task samples	20
Outer loop optimization iterations	1000

.

The training is carried out on a desktop computer with an operating system of Windows 10, the CPU is Intel (R) Core (TM) i7-6700 CPU @ 3.40GHz, and the memory is 16 GB.

4.3. Simulation Result

Using the policy initialized randomly as a baseline policy, compare the generalization ability on new tasks with that of the policy initialized using MAML meta-reinforcement learning. In order to verify the effectiveness of MAML meta-reinforcement learning, this paper samples the following two tasks from the task distribution for analysis.

4.3.1. Task 1

Take the task of controlling PCM to reach the target point (0, 80, 740) mm as the new task. To evaluate the generalization performance, both the policy initialized using the MAML meta-reinforcement learning and the baseline policy are trained to adapt to new tasks using the REINFORCE algorithm, and their generalization performances are compared. The training processes for both policies employ the same hyperparameters. The training result is shown in Figure 4.

As illustrated in Figure 4, when faced with a new task, the policy initialized with MAML meta-reinforcement learning reaches convergence in the excepted reward after 60 updates with a training time of 5 min. The baseline policy reaches convergence in the excepted reward after 180 updates with a training time of 15 min. By comparison, it is evident that the MAML meta-reinforcement learning algorithm has learned effective initial weights for the policy network from historical experiences. These initial weights enable the policy to adapt to new tasks with minimal updates, improving the adaptive capability for end position control of the PCM.

After the training of the policy initialized by MAML meta-reinforcement learning is completed, we use the trained policy to control the PCM. The position changes and the position error changes of the PCM are shown in Figure 5 and Figure 6, respectively.

The position error is (2.77, −2.82, −2.78) mm, and the maximum error is within 1.4%. Thus, the policy initialized with MAML meta-reinforcement learning can effectively control the PCM to reach the new target point after training.

To compare the performance with other reinforcement learning algorithms, the control policies for PCM are trained using the PPO algorithm, the Actor–Critic algorithm, and the PILCO algorithm. The hyperparameters of the PPO algorithm are as follows: the clipping parameter $\epsilon =0.2$, the learning rate for the policy network $l_a=1\times {10}^{-4}$, and the learning rate for the value network $l_c=5\times {10}^{-3}$. Both the policy network and the value network have one hidden layer, and the number of neurons in the hidden layer is 256. The hyperparameters of the Actor–Critic algorithm are as follows: the learning rate for the policy network $l_a=1\times {10}^{-4}$, and the learning rate for the value network $l_c=5\times {10}^{-3}$. Both the policy network and the value network have one hidden layer, and the number of neurons in the hidden layer is 256. After the training of the policy, we use the trained policy to control the PCM. The position changes and position error changes of the PCM are shown in Figure 7 and Figure 8, respectively.

The training times and errors of these algorithms are shown in

Table 2. Comparison of training results in task 1.

Algorithm	Training Time (min)	Maximum Absolute Error (mm)
MAML meta-reinforcement learning	5	2.82
PPO	28	41.12
Actor–Critic	17	23.97
PILCO	47	4.6

.

From Table 2, it can be seen that the policy initialized with MAML meta-reinforcement learning requires the shortest training time and achieves higher control precision during new task training. Compared to other algorithms, this approach enhances both training speed and control accuracy.

4.3.2. Task 2

Take the task of controlling PCM to reach the target point (22, −166, 736) mm as the new task. To evaluate the generalization performance, both the policy initialized using the MAML meta-reinforcement learning and the baseline policy are trained to adapt to new tasks using the REINFORCE algorithm, and their generalization performances are compared. The training processes for both policies employ the same hyperparameters. The training result is shown in Figure 9.

As illustrated in Figure 4, when faced with a new task, the policy initialized with MAML meta-reinforcement learning reaches convergence in the excepted reward after 60 updates with a training time of 5 min. The baseline policy reaches convergence in the excepted reward after 180 updates with a training time of 15 min. By comparison, it is evident that the MAML meta-reinforcement learning algorithm has learned effective initial weights for the policy network from historical experiences. These initial weights enable the policy to adapt to new tasks with minimal updates, improving the adaptive capability for end-position control of the PCM.

After the training of the policy initialized by MAML meta-reinforcement learning is completed, we use the trained policy to control the PCM. The position changes and position error changes of the PCM are shown in Figure 10 and Figure 11, respectively.

The position error is (2.77, −2.82, −2.78) mm, and the maximum error is within 1.4%. Thus, the policy initialized with MAML meta-reinforcement learning can effectively control the PCM to reach the new target point after training.

To compare the performance with other reinforcement learning algorithms, the control policies for PCM are trained using the PPO algorithm, the Actor–Critic algorithm, and the PILCO algorithm. The hyperparameters of the PPO algorithm are as follows: the clipping parameter $\epsilon =0.2$, the learning rate for the policy network $l_a=1\times {10}^{-4}$, and the learning rate for the value network $l_c=5\times {10}^{-3}$. Both the policy network and the value network have one hidden layer, and the number of neurons in the hidden layer is 256. The hyperparameters of the Actor–Critic algorithm are as follows: the learning rate for the policy network $l_a=1\times {10}^{-4}$, and the learning rate for the value network $l_c=5\times {10}^{-3}$. Both the policy network and the value network have one hidden layer, and the number of neurons in the hidden layer is 256. After the training of the policy, we use the trained policy to control the PCM. The position changes and position error changes of the PCM are shown in Figure 12 and Figure 13, respectively.

The training times and errors of these algorithms are shown in

Table 3. Comparison of training results in task 2.

Algorithm	Training Time (min)	Maximum Absolute Error (mm)
MAML meta-reinforcement learning	5	4.9
PPO	20	34.8
Actor–Critic	25	25.9
PILCO	47	3.9

.

From Table 3, it can be seen that the policy initialized with MAML meta-reinforcement learning requires the shortest training time and achieves higher control precision during new task training. Compared to other algorithms, this approach enhances both training speed and control accuracy.

5. Experiment

5.1. Experimental Platform

Take the two-section PCM as the experimental object, The experimental platform is shown in Figure 14 and Figure 15. The setup included a power supply, the PCM, a camera, proportional valves, an I/O module, a PC, and an air compressor. The camera captured the position of the manipulator’s end effector, and the PC output control signals to the PCM based on this position. After receiving the control signals from the PC, the I/O module would drive the proportional valves to adjust the pressure of the pneumatic muscles, thereby driving the end effector of the pneumatic continuum manipulator to move.

5.2. Experimental Analysis

In the real world, the same initial and target positions as in Task 1 in the simulation were selected, that is, to control the PCM to reach the target point (40, 80, 740) mm.

Employing the sim-to-real transfer approach based on policy fine-tuning, the control policy was fine-tuned on the actual PCM. After fine-tuning, we used the method of sim-to-real transfer based on policy fine-tuning to retrain the control policy on the PCM. After training, the position changes and position error changes of the PCM are shown in Figure 16 and Figure 17, respectively.

As shown in Figure 16 and Figure 17, the policy can control the PCM to reach the target position, with maximum errors of 8 mm along the X-axis, Y-axis, and Z-axis. The maximum error is within 4%.

6. Conclusions

This paper applies meta-reinforcement learning methods to the position control of the PCM, using MAML meta-reinforcement learning to train the position control policy of the PCM. MAML meta-reinforcement learning dynamically adjusts control policy to adapt to different tasks by learning general initialization parameters of policy, thereby enhancing the adaptability of the policy to new control tasks. Finally, based on the experimental platform, the adaptive ability of MAML meta-reinforcement learning was verified. The results showed that the control policy trained by the MAML meta-reinforcement learning algorithm could complete the control task of the PCM with minimal training when facing new tasks, improving the adaptability of the control policy.