The Maintenance of Orbital States in a Floating Partial Space Elevator Using the Reinforcement Learning Method ^†

Abstract

A partial space elevator (PSE) is a multi-body tethered space system in which the main satellite, typically an ultra-large spacecraft or a space station in a higher orbit, is connected to a transport spacecraft in a lower orbit via a tether, maintaining orbital synchronization. One or more climbers can move along the tether driven by electric power, enabling cross-orbital payload transportation between the two spacecraft. The climbers’ motion significantly alters the main satellite’s orbital states, compromising its safe and stable operation. The dynamic coupling and nonlinearity of the PSE further exacerbate this challenge. This study aims to preliminarily address this issue by proposing a new mission planning strategy. This strategy utilizes reinforcement learning (RL) to select the waiting interval between two transfer missions, thereby maintaining the main satellite’s orbital motion in a stable state. Simulation results confirm the feasibility and effectiveness of the proposed mission-based method.

1. Introduction

A PSE is a multi-body tethered space system (TSS), with its overall structure depicted in Figure 1 [1]. The main satellite, which is typically a space station or ultra-large space structure, is located in a higher orbit [2] and utilizes a tether to anchor the end body, which is typically a transport spacecraft or space experimental platform in a lower orbit. The length of the tether can range from tens to thousands of kilometers, depending on the mission requirements and background. Climbers can move along the tether rapidly using electric power, enabling efficient cross-orbit payload transportation between the two spacecraft. The energy cost can be reduced to less than 60% of traditional rocket transportation means [3], and the unit cost of payload transportation is expected to be reduced to below 5% of traditional means [4]. Additionally, a PSE allows for flexible deployment and deconstruction by releasing or retrieving the tether, following environmental conditions and requirements. Here, it should be noted that in practical applications, tethers several tens of kilometers in length have been used for deep-sea exploration and mooring floats [5,6]. Currently, there are no applications or preparations of tethers that are thousands of kilometers long for ground missions. The weightless environment in which PSEs and space elevators will be located will be an ideal scenario for their future application [7].

In the past decade, studies in the field of tethered spacecraft have mainly focused on the precise modeling, dynamic calculations, and libration suppression of PSEs [8,9,10,11]. Works have made initial progress in addressing the challenges related to accurate modeling and computation and have proposed relatively comprehensive libration suppression methods. As research deepens, the way to deal with the orbital floating and oscillation of the main satellite caused by payload transportation is becoming a key challenge [12,13] that relates to previous work.

The dynamics and modeling of PSEs can include studies on three-body TSSs, which have moving masses. Lorenzini [14], Misra [15], Williams [16], Cohen [17], and Jung [17] et al. built dynamic models of three-body TSSs, and some dynamic characteristics have also been analyzed. In 2010, Woo and Mirsa [18] introduced the concept of the PSE. Then, the dynamic responses of a PSE to a climber’s motion were analyzed by Shi and Zhu et al. [9,19] using a simplified model. These responses were verified by Yu et al. [20] through ground experiments. To provide a more accurate dynamic model of a PSE, Li and Zhu [21,22] developed a nodal position finite element method-based high-fidelity model. This model incorporates most physical factors, such as tether elasticity and flexible motions. They found that the simplified model of a PSE is sufficiently accurate for describing libration and orbital motions. Additionally, Shi and Zhu [12,13] found that the climber’s movement along the tether significantly alters the main satellite’s orbit, which is noteworthy for maintaining the satellite’s orbital stability during payload transportation.

Stability control of PSEs is crucial, primarily focusing on suppressing libration. Consequently, many stabilization techniques for TSSs can be applied to PSEs. Super-twisting sliding mode control laws and MPC methods [23,24,25] have been investigated to mitigate the libration motion of TSSs, and these control methods are also applicable to PSEs. To achieve a stable configuration in the transfer period of PSEs, the prescribed performance law was considered in a dual-loop control scheme [26]. Optimal control strategies are also effective and desired to suppress the libration from the planning perspective, in general [27,28]. However, the calculation process is complex, and the calculation workload is great. These issues have adverse effects on online control. To deal with these issues, piece-wise optimal schemes [29] and cooperative game methods [1] have been considered. In summary, both the main satellite and the end body need to be controlled, but they are two issues for PSEs. For the control of the main satellite, the target is to maintain its orbital state, especially the orbital radius in a desired realm. For the end body, the target is to maintain its position in a desired realm relating to the main satellite. Since the end body is commonly a transporting spacecraft with high mobility, the control of end bodies is a common issue. However, for the main satellite, which is typically regarded as an ultra-large space structure with a very large mass, its orbital control is a complex issue that has to be considered based on multiple aspects.

In this work, we focus on the maintenance of the main satellite’s orbital states. The aforementioned approaches can maintain the orbital states of the main satellite. However, they require additional thrusters and have a limited adjustment range. Our previous research [12] demonstrated that varying the transportation start time, which corresponds to the initial state of the PSE, results in different orbital states after the transportation. This provides a new mission-based planning method for maintaining the orbital states of the main satellite using the exhaustion method. Its drawbacks include (i) its inability to quickly achieve mission planning in orbit with small step sizes and (ii) its omission of optimal and suboptimal solutions with larger step sizes. To solve these issues, in this study, we propose and implement a mission-based RL method by selecting a waiting interval between two transfer missions (one upward and one downward). This new mission-based planning strategy leverages the RL method. The planning policy is trained using the deep Q-network algorithm, where the waiting interval serves as the agent’s action. To effectively train the agent, we designed a new reward function that evaluates the main satellite’s stability after the climber’s transportation. The proposed method can maintain the orbit radius of the main satellite to a large extent without using thrust, thereby saving fuel for the precise orbit control of the main satellite that requires thrust in the future. The numerical simulation results demonstrate the effectiveness of the new mission-based method using RL.

This paper is organized as follows. In Section 2, we model the dynamics of a PSE. Subsequently, the problem is described, and an RL-based method based on the specificity of the problem is proposed in Section 3. The proposed mission-based method is validated by the case study in Section 4. Finally, in Section 5, the conclusions are given in summary, and prospects for future work are provided.

2. Mathematical Formulation

As illustrated in Figure 1, M, m₁, and m₂ represent the masses of the main satellite, climber, and end body, respectively. The PSE operates in an ideal central gravitational field with disturbances like solar radiation pressure and atmospheric drag being ignored. The main satellite, which is typically a large-mass space structure, is located in a higher orbit and utilizes a tether to anchor the end body in a lower orbit. The length of the tether can range from tens to thousands of kilometers, depending on the mission requirements and background. The climber can move along the tether rapidly using electric power, enabling efficient cross-orbit payload transportation between the main satellite and the end body.

Here, it should be noted that the out-of-plane motions of the PSE are neglected, as it can be safely decoupled from the in-plane motions due to the weak coupling nature between these two modes of libration. Thus, we focus on the motions in the orbital plane using an in-plane model.

The vectors r, r₁, and r₂ are the projections of the absolute position of the main satellite, the climber, and the end body in the orbital coordinate frame, respectively, such that

$$\begin{array}{l}r=r{e}_r\\ r_1=r-L_1\cos {\theta }_1{e}_r-L_1\sin {\theta }_1{e}_{\vartheta }\\ r_2=r_1-L_2\cos {\theta }_2{e}_r-L_2\sin {\theta }_2{e}_{\vartheta }\end{array}$$

(1)

where r denotes the orbital radius of the main satellite, $\vartheta$ is the true anomaly, L₁ and L₂, are the tether lengths, and ${\theta }_1$ and ${\theta }_2$ denote the libration angles of the climber and the end body, which are measured from vector r₀ to L₁ and L₂, respectively. The unit vectors $e_{\vartheta }$ and $e_r$ are in the directions of the main satellite’s orbital motion and radius, respectively.

Differentiating Equation (1) yields

$$v={\displaystyle \frac{dr}{dt}}+\omega \times r,\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}{v}_1={\displaystyle \frac{d{r}_1}{dt}}+\omega \times r_1,\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}{v}_2={\displaystyle \frac{d{r}_2}{dt}}+\omega \times r_2$$

(2)

where $\omega ={[\begin{array}{ccc}0& 0& \dot{\vartheta }\end{array}]}^T$ and $\dot{\vartheta }$ is the angular velocity of the true anomaly. We define the Lagrange equation as L = K − U, where potential energy (U) and kinetic energy (K) are

$$\begin{array}{l}U=-\mu \left({\displaystyle \frac{M}{\left|r\right|}}+{\displaystyle \frac{m_1}{\left|r_1\right|}}+{\displaystyle \frac{m_2}{\left|r_2\right|}}\right)\\ K={\displaystyle \frac{1}{2}}M{v}^T\cdot v+{\displaystyle \frac{1}{2}}{m}_1{v}_1^T\cdot v_1+{\displaystyle \frac{1}{2}}{m}_2{v}_2^T\cdot v_2\end{array}$$

(3)

The generalized coordinates are $\left(q_1,q_2,q_3,q_4,q_5,q_6\right)=\left({\theta }_1,{\theta }_2,L_1,L_2,r,\vartheta \right)$, such that the Lagrange equation can be written as

$${\displaystyle \frac{d}{dt}}{\displaystyle \frac{\partial L}{\partial {\dot{q}}_i}}-{\displaystyle \frac{\partial L}{\partial q_i}}=Q_i\left(i=1,2,3,4,5,6\right)$$

(4)

The generalized force $Q_i$ can be derived by

$$Q_i={\displaystyle \sum _{j=0}^2{{\Lambda }_j}^T\frac{\partial r_k}{\partial q_i}}\hspace{0.33em}\hspace{0.33em}(j=\hspace{0.33em}0,\hspace{0.33em}1,\hspace{0.33em}2)$$

(5)

where

$$\begin{array}{l}{\Lambda }_0=-T_1\cos {\theta }_1{e}_r-T_1\sin {\theta }_1{e}_{\vartheta }\\ {\Lambda }_1=\left(T_1\cos {\theta }_1-T_2\cos {\theta }_2\right)e_r+\left(T_1\sin {\theta }_1-T_2\sin {\theta }_2\right)e_{\vartheta }\\ {\Lambda }_2=T_2\cos {\theta }_2{e}_r+T_2\sin {\theta }_2{e}_{\vartheta }\end{array}$$

(6)

The dynamics of PSE can be derived using the Lagrange method [21] as

$${\ddot{\theta }}_1=-{\displaystyle \frac{3{\dot{\vartheta }}^2\sin 2{\theta }_1}{2}}+{\displaystyle \frac{T_1\sin {\theta }_1}{rM}}-{\displaystyle \frac{T_2\sin ({\theta }_1-{\theta }_2)}{L_1{m}_1}}-{\displaystyle \frac{2(\dot{\vartheta }+{\dot{\theta }}_1){\dot{L}}_1}{L_1}}\hspace{0.33em}+{\displaystyle \frac{2\dot{\vartheta }\dot{r}}{r}}$$

(7)

$${\ddot{\theta }}_2=-{\displaystyle \frac{3{\dot{\vartheta }}^2\sin 2{\theta }_2}{2}}+{\displaystyle \frac{T_1\sin {\theta }_1}{rM}}+{\displaystyle \frac{T_1\sin ({\theta }_1-{\theta }_2)}{(L_0-L_1)m_1}}+{\displaystyle \frac{2(\dot{\vartheta }+{\dot{\theta }}_2){\dot{L}}_1}{L_0-L_1}}+{\displaystyle \frac{2\dot{\vartheta }\dot{r}}{r}}$$

(8)

$${\ddot{L}}_1={\dot{\vartheta }}^2{L}_1-{\displaystyle \frac{\mu \cos {\theta }_1}{r}}-{\displaystyle \frac{G_1^1\cos {\theta }_1+G_1^2\sin {\theta }_1}{m_1}}-{\displaystyle \frac{T_1}{m_0}}-{\displaystyle \frac{T_1}{m_1}}+{\displaystyle \frac{\cos ({\theta }_1-{\theta }_2)T_2}{m_1}}+(2\dot{\vartheta }+{\dot{\theta }}_1)L_1{\dot{\theta }}_1$$

(9)

$$\begin{array}{l}{\ddot{L}}_2={\dot{\vartheta }}^2{L}_2+{\displaystyle \frac{G_1^1\cos {\theta }_2+G_1^2\sin {\theta }_2}{m_1}}-{\displaystyle \frac{G_2^1\cos {\theta }_2+G_2^2\sin {\theta }_2}{m_1}}-{\displaystyle \frac{T_1}{m_1}}-{\displaystyle \frac{T_2}{m_2}}\\ \hspace{1em}+(2\dot{\vartheta }+{\dot{\theta }}_2)L_2{\dot{\theta }}_2+{\displaystyle \frac{T_1\cos ({\theta }_1-{\theta }_2)}{m_1}}\end{array}$$

(10)

$$\ddot{r}=r{\dot{\vartheta }}^2-{\displaystyle \frac{\mu }{r^2}}-{\displaystyle \frac{T_1\cos {\theta }_1}{M}},\hspace{0.33em}\hspace{0.33em}\ddot{\vartheta }=-{\displaystyle \frac{2\dot{\vartheta }\dot{r}}{r}}-{\displaystyle \frac{T_1\sin {\theta }_1}{rM}}$$

(11)

$$\begin{array}{l}{G}_1^1=-{\displaystyle \frac{\mu m_1(r-L_1\cos {\theta }_1)}{{\left|r_1\right|}^3}},\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}{G}_1^2={\displaystyle \frac{\mu m_1{L}_1\sin {\theta }_1}{{\left|r_1\right|}^3}}\\ G_2^1=-{\displaystyle \frac{\mu m_2(r-L_1\cos {\theta }_1-L_2\cos {\theta }_2)}{{\left|r_2\right|}^3}},\hspace{0.33em}\hspace{0.33em}{G}_2^2={\displaystyle \frac{\mu m_1(L_1\sin {\theta }_1+L_2\sin {\theta }_2)}{{\left|r_2\right|}^3}}\end{array}$$

(12)

where µ is the Earth’s gravitational parameter and T₁ and T₂ are the tether tensions in L₁.

The tensions in tethers can be calculated by assuming that the total tether length L₀ is constant, such that

$$L_2=L_0-L_1,\hspace{0.33em}{\dot{L}}_2=-{\dot{L}}_1,\hspace{0.33em}{\ddot{L}}_1=-{\ddot{L}}_2$$

(13)

Substituting Equation (13) into Equations (9) and (10) yields

$$\begin{array}{ll}{T}_1=& m_0\{r^2{G}_1^1\left[m_2\sin {\theta }_2\sin ({\theta }_1-{\theta }_2)-m_1\cos {\theta }_1\right]\\ & -r^2{G}_1^2\left[m_1\sin {\theta }_1+m_2\cos {\theta }_2\sin ({\theta }_1-{\theta }_2)\right]\\ & +m_1\{-\mu (m_1+m_2)\cos {\theta }_1+r^2\{(m_1+m_2)\left[L_1{(\dot{\vartheta }+{\dot{\theta }}_1)}^2-{\ddot{L}}_1\right]\\ & -\cos ({\theta }_1-{\theta }_2)\sin {\theta }_2{G}_2^2-\cos ({\theta }_1-{\theta }_2)\cos {\theta }_2{G}_2^1\\ & +m_2\cos \left({\theta }_1-{\theta }_2\right)\left[L_2{(\dot{\vartheta }+{\dot{\theta }}_2)}^2+{\ddot{L}}_1\right]\}\}\}/A\end{array}$$

(14)

$$\begin{array}{ll}{T}_2=& \{-2r^2{G}_2^1{m}_1(m_1+m_0)\cos {\theta }_2-2r^2{G}_2^2{m}_1(m_1+m_0)\sin {\theta }_2\\ & +m_2\{2r^2{G}_1^1\left[m_0\sin {\theta }_1\sin ({\theta }_1-{\theta }_2)+m_1\cos {\theta }_2\right]\\ & +\sin {\theta }_2\{2r^2{G}_1^2(m_1+m_0{\cos }^2{\theta }_1)+2m_1{m}_0\{r^2\left[L_1{(\dot{\vartheta }+{\dot{\theta }}_1)}^2-{\ddot{L}}_1\right]-\mu \cos {\theta }_1\}\}\\ & -2m_0\cos {\theta }_1\cos {\theta }_2\{r^2{G}_1^2\sin {\theta }_1+m_1\{\mu \cos {\theta }_1-r^2\left[L_1{(\dot{\vartheta }+{\dot{\theta }}_1)}^2-{\ddot{L}}_1\right]\}\}\\ & +2r^2{m}_1(m_1+m_0)\left[L_2{(\dot{\vartheta }+{\dot{\theta }}_2)}^2+{\ddot{L}}_1\right]\}\}/(2r^{2}A)\end{array}$$

(15)

where $A=m_1(m_1+m_2)+m_0\left[m_2\hspace{0.33em}{\sin }^2({\theta }_1-{\theta }_2)+m_1\right]$.

3. Maintenance of the Main Satellite’s Orbital States Using the RL Method

3.1. Problem Formation

As shown in Figure 2, the movement of the climber along the tether causes the orbital radius (r) to fluctuate rather than remain constant, and the $\left|\dot{r}\right|$ ≠ 0 after the transfer mission. Furthermore, large values of $\left|\dot{r}\right|$ will lead to significant fluctuations in r, which is detrimental to the safety of the main satellite. Δr = r − r₀ denotes the changing magnitude of r, where r₀ is the initial orbital radius of the main satellite. One reasonable condition after one transfer mission is that the Δr is small (see the blue lines in Figure 2), and the orbital states after the transfer mission in (b) are better than those in (a). The system parameters in this case are shown in

Table 1. System parameters and initial states of the PSE.

Parameters	Values
Orbital radius of the main satellite, r (m)	7.1 × 106
Initial True anomaly angular, $\vartheta (0)$(rad)	0
Mass of the main satellite (net weight), M (kg)	50,000
Mass of climber (net weight), m1 (kg)	100
Mass of the payload (kg)	400
Mass of end body, m2 (kg)	1000
Fixed total tether length, L0 (m)	2 × 104
Initial libration angle and angular velocity, (${\theta }_1\left(0\right)$, ${\dot{\theta }}_1\left(0\right)$)	(0, 0)
Initial libration angle and angular velocity, (${\theta }_2\left(0\right)$, ${\dot{\theta }}_2\left(0\right)$)	(0, 0)
Initial distance between the climber and the main satellite, L1 (m)	19,500
Reward function parameter, λ	100
Adaptable average orbital radius changing magnitude $\mathsf{\Delta }\overline{r}$(m)	100

. In summary, after one transfer mission, the main satellite’s orbital radius fluctuates around its initial value over time with an obvious changing magnitude of r. This changing magnitude may either decrease or increase after the next transfer mission depending on the starting time of the next transfer mission.

Then, the problem is how to select such a mission starting time so as to minimize the changing magnitude of r after one transfer mission as much as possible. Such selection is difficult. This is because the required input in the mission plan is the mission starting time, a time point. Its dynamic impact is caused by the movement of the climber within a transfer period after that mission starting time. It is difficult to describe the relationship between the mission starting time and the changing magnitude of r after the transfer period in dynamics. As a result, existing optimization theories are not suitable for optimizing the selection of the mission starting time.

Considering the situations above, one desired strategy is the deep RL method. The RL method for the maintenance of the orbital states of a PSE involves (i) utilizing the powerful fitting capabilities of deep neural networks to establish the mapping relationship between the departure time and the main satellite’s orbital state after the transfer mission and (ii) utilizing a trained agent to make decisions. In summary, deep neural networks can be used to address the problem (i), but they must also make decisions on mission execution to provide an action, which is why RL methods are needed. The powerful representation ability of deep neural networks makes them a powerful tool for state representation and function approximation in RL.

3.2. RL-Based Mission Planning Method

This work employs the RL method to address the complex dynamics of the PSE by training an agent (controller) using the Deep Q-Network (DQN) learning algorithm. Based on the problem formulation, mission planning for the PSE in this study involves selecting the optimal mission starting time. This can be facilitated by determining the length of the waiting interval h from the end time point of the previous transfer mission. Then, the action can be defined in a discrete dataset. Once the action is completed, the episode ends. As a result, the length of each episode is equal to h. The initial states of the agent for each episode are the same. Then, the DQN learning algorithm framework can be summarized as follows (see Figure 3). First, the model, training algorithm, agent (action), and experience replay memory are initialized and set. Interaction data and storage data are collected, then the training begins for the preset episode numbers. The detailed process is as follows:

(1) Learning environment: The DQN learning algorithm is designed to solve decision problems based on Markov decision processes (MDPs). The learning agent interacts with an environment, such that:

The state space is defined as the generalized coordinates of the PSE, including θ₁, ${\dot{\theta }}_1$, θ₂, ${\dot{\theta }}_2$, L₁, ${\dot{L}}_1$, r, $\dot{r}$, ϑ, and $\dot{\vartheta }$.

The action is defined as the waiting interval h.

The reward function determines the immediate feedback that guides the agent’s learning. The agent aims to maximize the cumulative reward over time, and the specific design of the reward function in this work is defined as

$$Reward\hspace{0.33em}=\hspace{0.33em}\Delta \overline{r}-\left(\max {\left|r\right|}_n-\min {\left|r\right|}_n\right)-\lambda \max {\left|\dot{r}\right|}_n$$

(16)

where $\mathsf{\Delta }\overline{r}$ is an average orbital radius changing magnitude which can be set by the user, while λ is a constant parameter. λ > 0 is a constant parameter that is used to penalize excessive orbital radius change rates. |r|_n and ${\left|\dot{r}\right|}_n$ denote the orbital radius of the main satellite in n test orbital periods. Here, the test orbital period is defined as the orbital period after the transfer mission (see Figure 2b). In such orbital periods, the climber’s transportation is completed, and the main satellite’s radius keeps fluctuating. $\max {\left|r\right|}_n-\min {\left|r\right|}_n$ denotes the amplitude of change in the main satellite’s orbital radius over n test orbital periods after the transfer mission. Here, it should be noted that in this work, each episode comprises only one action.

(2) Construct the Q-network: The network uses generalized coordinates as the input and generates Q-value estimates for each possible action.

(3) Initialize network parameters: The biases and weights of the deep Q-network are initialized.

(4) Select an action: Based on the estimations and current state provided by the deep Q-network, an action is chosen using an ε-greedy strategy to balance exploration and exploitation.

(5) Execute the action: The chosen action is performed, and the environment will return a reward and the next state.

(6) Record experiences: Current states (θ₁, ${\dot{\theta }}_1$, θ₂, ${\dot{\theta }}_2$, L₁, ${\dot{L}}_1$, r, $\dot{r}$, ϑ, $\dot{\vartheta }$), actions (Δt), rewards, and next states are stored as experience tuples.

(7) Experience replay: A batch of experience tuples are randomly sampled from the experience storage to train the deep Q-network.

(8) Update Q-values: The sampled experiences are utilized to adjust the parameters of the deep Q network, aiming to reduce the disparity between the predicted Q-values and the target Q-values.

(9) Compute target Q-values: Target Q-values for updating the deep Q-network are determined. These target Q-values include the highest Q-value for the subsequent state combined with the current reward.

(10) Train the network: Backpropagation and gradient descent are applied to reduce the discrepancy between the predicted Q-values and the target Q-values, thereby updating the parameters of the deep Q network.

(11) Steps (4) to (10) are repeated until a stopping condition or convergence is reached.

By iteratively updating the parameters of the deep Q-network, the DQN learning algorithm gradually improves the estimation of Q-values and discovers the optimal policy for RL missions.

4. Numerical Simulation and Discussion

The proposed mission-based method is validated by the case study with the following system parameters and the initial states of the PSE in Table 1.

The payload is assumed to be 400 kg and the climber’s net weight is 100 kg, such that, in the upward transportation, M = 50,000 kg and m₁ = 500 kg. In the waiting interval, M = 50,500 kg, and in the downward transfer period, m₁ = 100 kg and M = 50,400 kg. The speed function of the climber is designed as a sine form, ${\dot{L}}_1=4.9\pi \sin \left(\pi t/2000\right)$. The mission plan aims to find an optimal waiting interval h to minimize Δr after the downward transfer mission. The policy is trained for 500 episodes. The hyperparameters of the DQN are shown in

Table 2. DQN hyperparameters.

Parameters	Values
Learning rate	0.01
Gradient threshold	inf
Discount factor	0.99
Batch size	100
Experience buffer length	1000

.

In this work, the ε-greedy exploration strategy is used with an ε decay of 0.005, and a minimum ε set at 0.01, where ε is the exploration factor. Long short-term memory is adopted to train the recurrent neural network with two fully connected layers employed with 40 hidden nodes, respectively. In the enhancement of the learning process, rectified linear units are incorporated between each layer. Adam is used as the optimizer. In the training process, the episode step is 1/36 of the orbit, which equals the step of h in the discrete dataset, and the dynamic time step in the RK-4 is 0.001 s. Numerical simulations are performed using MATLAB, specifically the RL App. Here, it should be noted that by using the Reinforcement Learning Designer App in MATLAB, the simulation can be replicated using the given parameters in Table 1 and Table 2.

The results of the numerical simulations are presented in Figure 4, Figure 5, Figure 6 and Figure 7. The variation in the episode reward with the training process throughout the entire learning process is shown in Figure 4. The first 150 episodes mainly represent the stages of algorithm random exploration and the accumulation of learning samples, corresponding to low reward values, and most of the tasks in the generated plan cannot be completed. As the learning process progresses, the RL method (using the DQN algorithm) gradually summarizes the characteristics of the planning model and obtains a higher return on the reward value. After 200 episodes of learning, the algorithm converges, and the reward value remains stable, at approximately 128.79.

The trained agent offers the policy to decide how long the PSE should wait for the next transfer mission. Figure 5 and Figure 6 show the effects of the obtained policy. The waiting interval after an upward transfer mission begins at 2000 s. Here, it should be noted that the randomness of the obtained policy’s decisions always exists due to the complexity of the corresponding problem in terms of dynamics and the limited number of training episodes. Thus, a relatively small number of simulations are still needed to demonstrate the results. Thus, the agent gives 10 waiting intervals to achieve the highest reward (see Figure 5). All reward values are near 130 with the waiting interval h = 3.67 orbits = 19,674.7 s.

Figure 6 shows the state of the PSE following the planned mission. The libration angles fluctuate in the mission period by a magnitude of 0.6 rad in the beginning, and then both θ₁ and θ₂ change by magnitudes less than 0.3 rad due to the lack of libration suppression (see Figure 6a). The tensions of T₁ and T₂ are greater than zero in the simulation period, which means that the tethers are straight (see Figure 6c). This matches the assumptions in Section 2. The angular velocity in Figure 6d shows that the main satellite runs around the Earth with a nearly constant angular velocity, even though the orbital radius changes in the scale of 100 m. Figure 6e,f show the tether length change for L₁ and the climber speed ${\dot{L}}_1=4.9\pi \sin \left(\pi t/2000\right)$. This function ensures that the climber moves along the tether following the prescribed distance and period with the initial and final speeds being zero, which matches the engineering condition well. The upward transfer mission leads to $\max \left|r\right|-\min \left|r\right|>260m$, as seen in the curve of Δr in the period of 0.4–3.6 orbits in Figure 6b. Then, the downward transfer of the climber following the planned waiting interval reduces the amplitude change of the orbital radius to 60m. This means that the proposed orbital state maintenance method is effective.

Figure 7 shows that the waiting interval generated by the proposed RL method reduces the magnitude of orbital radius change after the mission by over 88% compared to the case in which the waiting interval is one orbital period, comparing the blue lines in (a) and (c). Figure 7b shows the result with the waiting interval obtained by exhaustion with the step of 5 min, which is small enough for the exhaustion of a PSE. Although the main satellite’s orbital state is kept near zero, the amplitude of the orbital radius is 70 m, which is greater than that in the case in which the waiting interval is generated by the RL method. This is because the searching step of the exhaustion is not small enough to achieve the optimal waiting interval.

5. Conclusions

This work presents an RL method to address the maintenance of the orbital states of the main satellite of a PSE based on the requirements and constraints of transportation missions. This new method focuses on planning the waiting interval between two transfer missions from a mission planning perspective in order to minimize the amplitude changes of the orbital radius of the main satellite after completing a transfer mission. The deep Q-network algorithm is employed to train the mission planning policy using a novel reward function. The simulation results demonstrate the following findings:

(i) The proposed mission-based RL method effectively maintains the orbital states of the main satellite.

(ii) With the transportation mission and the proposed actions, even a small number of training episodes can produce effective agents.

(iii) Compared to the exhaustion method, the RL method enables quick decision-making with agents trained offline.

Furthermore, based on the simulation results and conclusions in this work, it can be found that it is necessary to conduct targeted and in-depth research on the dynamic evolution mechanisms of PSEs. Since the overlapping time scales of transfer missions and orbits make the system dynamics mechanisms of the climber transportation mission process unclear, it is difficult to establish effective mapping between “mission orbits” through deep neural networks. This leads to the task sequence planning method based on RL being poorly generalized.