Good Match between “Stop-and-Go” Strategy and Robust Guidance Based on Deep Reinforcement Learning

Abstract

This paper deals with the guidance problem of close approaching small celestial bodies while autonomously navigating with an optical camera. A combination of a deep reinforcement learning (DRL)-based guidance method and a “Stop-and-Go” (SaG) strategy is here proposed to increase the mission adaptability. Firstly, a robust guidance strategy optimizing fuel consumption and angle-only navigation (AON) observability is trained by DRL. Secondly, the SAG strategy is designed to introduce the mission adaptability and further improve the AON observability. Thirdly, a good match between the SAG strategy and the DRL-based robust guidance is demonstrated. The proposed method was tested in a typical R-bar approaching scenario. Then, the mission adaptability with an onboard application was successfully verified, investigating the policy performance with SAG.

1. Introduction

Mission adaptability is a crucial but challenging capability for small celestial body exploration. In proximity to small celestial bodies, the environment is extremely harsh and unknown. The close approach phase to small celestial bodies faces many uncertainties and even emergencies. In current small celestial body missions [1,2], the close approach process is usually highly supervised from the ground [3,4], and the ground is always ready to intervene to ensure mission safety.

However, after the close approach process is interfered with by the ground, the original approach trajectory is interrupted, and the original plan is abandoned. A re-approach process and trajectory needs to be re-planned on the ground. This kind of inflexibility reduces the efficiency of temporary task adjustment, and may make it impossible to continue to achieve the set goals after the suspension caused by emergencies. It is urgent to design a spacecraft navigation and guidance strategy to enhance mission adaptability, which will have a promising prospect in small celestial body missions.

The close approach phase sends the spacecraft to the proximity of asteroids, and then the spacecraft can carry out complex operation tasks. For example, Hayabusa2 started a descent operation from a home position where the distance from the near-Earth asteroid (NEA) Ryugu was 20 km after Hayabusa2 had arrived at the proximity of the asteroid [5,6]. Therefore, the close approach phase is vital for mission design [2,7,8]. Specifically, the requirement of the guidance, navigation, and control (GN & C) system is presented very stringently, especially in terms of accuracy and robustness. The terminal accuracy and the robustness to various uncertainty are essential to ensure mission safety and success [4].

“Robustness” of trajectory or guidance is not a new concept. Research on stochastic optimal control [9,10] and stochastic dynamic programming [11,12] have been very hot. There is fruitful research on designing robust trajectory, such as chance-constrained optimal control [13,14], stochastic differential dynamic programming (SDDP) [11,12], evidence theory [15], and belief-based transcription procedures for stochastic optimal control problems (SOCPs) [9,10].

In the past few years, machine learning has been very popular in the application of spacecraft guidance, navigation, and control (GN & C). Izzo et al. summarized the development trend of machine learning in the field of spacecraft guidance dynamics and control and predicted future research in 2019 [16]. Supervised learning, the oldest branch of machine learning, has also shown its robustness to uncertainties [17]. However, under uncertain environments and complex constraints, a sufficiently large data set of experts is usually required for supervised learning, which needs careful and intricate design to avoid overfitting. Reinforcement learning (RL) has gradually gained more attention [18], which can autonomously learn control strategies through its interaction with the environment. The trajectory design problem is modeled as a time-discrete Markov decision process (MDP). The learning procedure is realized by maximizing a user-defined function, that is, a reward function. The policy gradient is used to continuously improve the policy, which is used as a baseline as this was later improved through the years. Policy-gradient (PG) algorithms extend deep Q-learning (a finite set of actions) to the field of continuous control (continuous state and action spaces) [19]. To alleviate the problem of the large variance of a cumulative reward, the actor–critic (AC) method and the advantage actor–critic (A2C) method improve the policy-gradient algorithms based on approximating the Q-function using a second DNN and approximating the advantage function using a generalized advantage estimator [20]. The proximal policy optimization (PPO) algorithm further stabilizes the update of the policy by introducing a clipped function [21]. PPO has demonstrated state-of-the-art performances on many benchmark RL problems. PPO is widely used partially because it does not need any ad-hoc reward shaping. In recent years, RL has been applied to spacecraft GN & C, such as low-thrust trajectory design [22,23,24,25,26], formation flying [27], rendezvous or intercept guidance [28,29,30,31,32,33], asteroid hovering [34,35], planetary landing [36,37,38,39,40,41], etc. Among them, reinforcement meta learning based on recurrent neural network is used to solve the problems of partially observable Markov decision process (POMDP) and policy extrapolation in the context of continuing learning [37,42,43]. Deep convolutional neural network is used to map the raw observations, such as raw images, to a true inner state [32,40], which is also a means to solve the POMDP.

Line of sight (LOS) navigation and AON are heated topics in noncooperative rendezvous [44], and are also very popular in small celestial body missions, especially those based on CubeSat. On the one hand, AON only needs optical sensors, which can reduce the cost and weight. On the other hand, AON is completely passive and self-contained, working over a wide range of distances. These advantages of AON make it more suitable than microwave radar, laser radar, laser rangefinder, and other sensors. However, AON has a well-known problem of weak range direction observability, and it is necessary to improve the range direction observability by incorporating known orbital maneuvers into the navigation filter [45,46,47]. Several optimal single-step rendezvous guidance methods have been proposed together with different definitions of AON observability [46,47]. Moreover, an online guidance framework of multiple maneuvers to obtain optimal observability has also been proposed [45], which is similar to model predictive control and needs to be solved multiple times online [48,49].

However, these proposed rendezvous guidance methods are all carried out in a determined manner by building and then solving a deterministic optimal control problem. The robust guidance strategy based on DRL is also usually trained under a fixed number of maneuvers [31,33]. Once an optimal guidance or rendezvous trajectory is obtained, the number of maneuvers is fixed. If the number of maneuvers needs to be changed, the problem needs to be solved again. This is one of the reasons for the mentioned inadaptability problem.

We find that the “Stop-and-Go” (SaG) strategy has the ability to increase the number of orbital maneuvers while basically maintaining the geometry of the original trajectory [3,50]. The basic idea of SaG is as follows [3,50]: a breaking to zero followed by a new impulsive maneuver is performed at a middle waypoint, which is intended to further assist the AON observability in the range direction. However, differential-corrective guidance (fixed-time-of-arrival) is employed in original papers [3,50], which can be more sensitive to various uncertainties.

This paper proposes a close approach guidance method with more robustness and adaptability. A good match between SaG strategy and robust DRL-based guidance is used. Specifically, in order to improve the AON observability, the observability term is introduced into the reward function. A flexible SaG strategy is designed to freely insert the SaG point on the close approach trajectory. A framework combining the SaG strategy and the DRL-based guidance is proposed. The advantages of this framework are:

(1)

The spacecraft can stop at any time during the process of approaching the asteroid to respond to an emergency or avoid hazards, giving the approach process more flexibility;

(2)

The original robust guidance can still work after the SaG point without the need to solve the guidance policy again, and therefore improve the real-time performance;

(3)

It can increase the AON observability in the range direction and therefore ensure the terminal accuracy of rendezvous guidance.

The structure of this paper is organized as follows. First, a robust guidance method with enhanced AON observability based on DRL is introduced in Section 2. Then, Section 3 introduces the SaG strategy and the framework combining the SaG strategy and the DRL-based guidance. Then, some simulation results in a typical R-bar approaching scenario are given in Section 4, and the mission adaptability with an onboard application is verified. Finally, Section 5 concludes with some interesting findings.

2. Robust Guidance Method with Enhanced AON Observability Based on DRL

2.1. Close Approach Phase

The reference task is an asteroid close approach rendezvous mission. For simplification, the spacecraft and the asteroid are assumed to follow a Kepler dynamic model under the sole influence of the sun’s gravity. This assumption holds true for the spacecraft when the gravity of the asteroid and the solar radiation pressure (SRP) are very small compared to the gravity difference of the sun, which means that the asteroid is very small in size and mass (like with a diameter smaller than 5 m), and that the area–mass ratio of the spacecraft is very small. The gravity acceleration due to the small body and the SRP acceleration can be considered in the dynamic model for a more general scenario, which will be included in future work. To describe the relative motion between a spacecraft and an asteroid, the coordinate systems shown in Figure 1 are established, including the asteroid Hill rotation coordinate system $O_{a}xyz$ and the heliocentric inertial coordinate system $O_{s}XYZ$. The spacecraft and asteroid can be regarded as points.

The center $O_a$ is at the barycenter of the asteroid. $O_{a}xyz$ rotates at the orbital angular velocity $\mathit{\omega }$ relative to $O_{s}XYZ$. Its basic plane is the asteroid orbital plane, and $O_{a}xyz$ is defined as follows:

(1)

The x axis is along the orbit radius.

(2)

The z axis follows the angular momentum direction of the asteroid’s orbit.

(3)

The y axis completes the right-handed coordinate system and is perpendicular to the x axis in the orbital plane.

The unit vector corresponding to the asteroid Hill rotation coordinate system $O_{a}xyz$ is $\left\{i_x, i_y, i_z\right\}$, and the unit vector corresponding to the heliocentric inertial coordinate system $O_{s}XYZ$ is $\left\{i_X, i_Y, i_Z\right\}$.

The relative position vector $\mathit{r}={\left[\begin{array}{ccc}x& y& z\end{array}\right]}^T$ is defined in the asteroid Hill rotation coordinate system, as shown in Figure 1. The spacecraft state $\mathit{x}$ is described by the relative position $\mathit{r}={\left[\begin{array}{ccc}x& y& z\end{array}\right]}^T$ and its time derivative $\mathit{v}=\dot{\mathit{r}}={\left[\begin{array}{ccc}\dot{x}& \dot{y}& \dot{z}\end{array}\right]}^T$, that is, $x={\left[\begin{array}{cc}{\mathit{r}}^T& {\mathit{v}}^T\end{array}\right]}^T$. The state $\mathit{x}$ is propagated by means of the orbital relative motion equation. The Clohessy–Wiltshire (C–W) equation can be used under the assumption of a circular orbit and short rendezvous times [38], and the Tschauner–Hempel (T–H) equation [51] can be used under the assumption of an elliptical orbit. After orbital maneuver $\mathit{u}$, the relative motion equation can be written as follows:

$$\mathit{x}\left(t\right)=\mathrm{\Phi }\left(t\right){\mathit{x}}_0+G\left(t\right)\mathit{u}$$

(1)

Equation (1) is defined in the asteroid Hill rotation coordinate system, where ${\mathit{x}}_0$ is the state vector at the initial maneuver time, and x is the state vector at time $t$ after an impulse maneuver. $\mathrm{\Phi }\left(t\right)$ is the state transition matrix, and the specific form of $\mathrm{\Phi }\left(t\right)$ depends on which of the C–W equation [47] or the T–H equation [52] is used. Since 2016HO3 (with the orbital eccentricity larger than 0.1) is adopted as the object asteroid in simulations, the T–H equation is chosen in this paper. For the derivation of the T–H equation and the detailed expression of $\Phi \left(t\right)$, please refer to Refs. [51,52].

The input matrix $G\left(t\right)$ in the case of an impulse maneuver is defined as Columns 4–6 of $\mathrm{\Phi }\left(t\right)$, that is,

$$G\left(t\right)=\mathrm{\Phi }\left(t\right)\left[1:6,4:6\right]=\left[\begin{array}{c}{G}_r\\ G_v\end{array}\right]$$

(2)

$$\mathrm{\Phi }=\left[\begin{array}{c}{\mathrm{\Phi }}_r\\ {\mathrm{\Phi }}_v\end{array}\right]$$

(3)

2.2. Markov Decision Process (MDP)

First, the general mathematical form of the discrete-time MDP is briefly introduced. The MDP is generally composed of state, observation, action, state transition, and reward. Specifically, an MDP is discretized into N discrete time steps, $\left[t_k,t_{k+1}\right],k\in \left[0,N-1\right]$. Let ${\mathit{s}}_k\in S\subset {ℝ}^n$ be the state vector of the spacecraft’s relative motion at time $t_k$, where $S$ is the state space. Generally, ${\mathit{s}}_k$ cannot be directly obtained by the controller, but be estimated or observed as an observation through a navigation filter. Let ${\mathit{o}}_k\in O\subset {ℝ}^m$ be the observation of the relative motion state ${\mathit{s}}_k$ at time $t_k$, where $O$ is the observation space. In the scenario of close approaching small celestial bodies, especially those with a diameter of less than 5 m, AON is one of desired options because AON only needs optical sensors and can work over a wide range of distances. Since adding the complete AON process to the DRL will greatly slow down the training speed, we decided not to use the real AON process to obtain ${\mathit{o}}_k$. On the contrary, the direct state observation is assumed. Specifically, the modeling of the state observation is based on introducing a manually designed noise inspired by the prior knowledge of AON performance, rather than a fixed Gaussian distribution additive noise. This will be explained in more detail in Section 2.3. Let ${\mathit{a}}_k\in A\subset {ℝ}^l$ be the control command defined as the output of the state feedback controller at time $t_k$, where $A$ is the action space. Let $\pi :O\to A$ be the state feedback control policy; then, ${\mathit{a}}_k=\pi \left({\mathit{o}}_k\right)$. Let $f:{\mathit{s}}_k,{\mathit{a}}_k\to {\mathit{s}}_{k+1}$ be the state transition function of the system, which can be described by a discrete-time CW equation. Let $R_k\in ℝ$ be the reward obtained during the transition from ${\mathit{s}}_k$ to ${\mathit{s}}_{k+1}$ under ${\mathit{a}}_k$. A discount factor $\gamma \in \left[0,1\right]$ is used to describe the fading of the current instantaneous reward’s importance to the future. For instance, let $\gamma$ be close to 1 when it is very important in the long run, and let $\gamma$ be close to 0 when it is only important in the short term. Let $e=\left\{\left({\mathit{s}}_0,{\mathit{a}}_0\right),\cdots ,\left({\mathit{s}}_{N-1},{\mathit{a}}_{N-1}\right)\right\}$ be one round of the trajectory episode obtained according to state feedback control strategy $\pi$. The objective of RL is to solve the optimal state feedback control strategy ${\pi }^{*}$ by optimizing a designed performance index, which is the sum of the expected value of the sum of $R_k$ along trajectory episode $e$ with discount factor $\gamma$, that is,

$$J=\underset{\tau ~\pi }{𝔼}\left[{\displaystyle \sum _{k=0}^{N-1}{\gamma }^k{R}_k}\right]$$

(4)

The above MDP is used to model a spacecraft close approach rendezvous. Considering a rendezvous mission with fixed time $t_f$, the rendezvous is discretized into N time steps uniformly with step size $\Delta T=t_f/N$. Then, state ${\mathit{s}}_k,k\in \left[0,N\right]$ is defined as the relative motion state $\mathit{x}$ at time $t_k=k\cdot \Delta T$, that is,

$${\mathit{s}}_k=\mathit{x}\left(t_k\right)\in {ℝ}^6$$

(5)

If impulse maneuver $\mathit{u}$ in Equation (2) occurs at time $t_k=k\cdot \Delta T$ and state $\mathit{x}$ is considered at time $t_{k+1}=\left(k+1\right)\cdot \Delta T$, then Equation (2) can be used to describe the transition of the state between two pulses. In this case, ${\mathit{u}}_k$ is the action ${\mathit{a}}_k$ taken at time $t_k$, and Equation (1) can be reformed as

$${\mathit{x}}_{k+1}=\mathrm{\Phi }{\mathit{x}}_{{}_k}+G{\mathit{u}}_k,k=0,1,\cdots ,N-1$$

(6)

where ${\mathit{x}}_{{}_k}\equiv \mathit{x}\left(t_k\right)$, $\mathbf{\Phi }\equiv \mathbf{\Phi }\left(\Delta T\right)$, $G\equiv G\left(\Delta T\right)$, and $\Delta T$ are omitted if there is no ambiguity. $\mathbf{\Phi }$ and $G$ are both constant matrixes.

At final time step $t_N=t_f$, according to the terminal velocity constraint and the maximum thrust constraint, the final thrust ${\mathit{u}}_N$ is calculated as follows:

$${\mathit{u}}_N=\min \left(\left|{\mathit{v}}_f-{\mathit{v}}_N\right|,\Delta v_{\max ,N}\right)\frac{{\mathit{v}}_f-v_N}{\left|{\mathit{v}}_f-v_N\right|}$$

(7)

where $\Delta v_{\max ,N}$ is the maximum thrust.

The spacecraft terminal state is defined as:

$${\mathit{r}}_f={\mathit{r}}_N$$

(8)

$${\mathit{v}}_f={\mathit{v}}_N+{\mathit{u}}_N$$

(9)

The indeterminable state observation at time $t_k$ is defined as:

$${\mathit{o}}_k={\left[{\mathit{r}}_k^T,{\mathit{v}}_k^T\right]}^T\in {ℝ}^6$$

(10)

2.3. Observation Uncertainty Analysis and Modeling

The relative position and velocity of a spacecraft cannot be obtained directly by a controller, so it is necessary to observe or estimate the relative position and velocity using a navigation filter. In AON, additional maneuvers will be used to assist convergence of the navigation filter [52]. In spacecraft rendezvous, there is a quantified minimum navigation accuracy rule, the 1% range rule, which is that the relative navigation accuracy needs to be guaranteed to be at least 1% of the relative distance [46]. When the navigation has not converged, the navigation error can be more than 10% of the relative range and can be less than 0.5% after convergence. In this study, instead of incorporating the full navigation process into the MDP and the training of RL, which can be time-consuming, we simplify it as a direct observation of the states with noise representing navigation errors.

Observation uncertainty is here modeled as a time-varying Gaussian additive noise, and the observation at time $t_k$ can be defined as:

$${\mathit{o}}_k={\mathit{x}}_k+{\mathit{\omega }}_{o,k}$$

(11)

$${\mathit{\omega }}_{o,k}=\left[\begin{array}{c}\delta {\mathit{r}}_{o,k}\\ \delta {\mathit{v}}_{o,k}\end{array}\right]\sim {\rm N}\left(0_{6\times 1},{\Sigma }_{\mathit{o},k}\right)\in {ℝ}^6$$

(12)

where ${\Sigma }_{o,k}=diag\left({\sigma }_{r,o,k}^2{I}_{3\times 3},{\sigma }_{v,o,k}^2{I}_{3\times 3}\right)$ is the covariance matrix. ${\sigma }_{r,o,k}$ and ${\sigma }_{r,o,k}$ are the standard deviations of velocity and position, respectively. More consideration will be given to ${\sigma }_{r,o,k}$ as follows:

$${\sigma }_{r,o,k}=\{\begin{array}{c}10\%\Vert \mathit{r}\Vert k\le k_c\\ {\sigma }_{pr} k>k_c\end{array}$$

(13)

where $\Vert \mathit{r}\Vert$ is the relative distance, and ${\sigma }_{pr}$ can be adjusted according to the performance of AON. $k_c$ is the time step where the navigation filter converges. In this paper, $k_c=2$.

2.4. Constraints Analysis and Modeling

The actual thrust limit of a thruster can be modeled using a thrust amplitude constraint. For an impulse maneuver, the thrust amplitude constraint can be described by the maximum impulse velocity. The amplitude of any impulse shall be less than or equal to the maximum value $\Delta v_{\max }$, i.e.,

$$\Vert {\mathit{u}}_k\Vert \le \Delta v_{\max }$$

(14)

During an asteroid close approach rendezvous, the approach direction of the spacecraft is constrained by a visibility cone. As shown in Figure 2, the projection of the visibility cone in the x–y plane shows how the motion of spacecraft is constrained by a cone half-angle of $\beta$. At time $t_k$, the three-dimensional approach visibility cone constraint can be expressed as:

$$\{\begin{array}{c}{y}_k+x_k\tan \beta \le 0\\ -y_k+x_k\tan \beta \le 0\\ z_k+x_k\tan \beta \le 0\\ -z_k+x_k\tan \beta \le 0\end{array}$$

(15)

It can be written in matrix form as:

$$B{\mathit{x}}_k\le 0_{4\times 1}$$

(16)

where

$${\rm B}=\left[\begin{array}{cccccc}\tan \beta & 1& 0& 0& 0& 0\\ \tan \beta & -1& 0& 0& 0& 0\\ \tan \beta & 0& 1& 0& 0& 0\\ \tan \beta & 0& -1& 0& 0& 0\end{array}\right]$$

(17)

For different approach directions, the inequalities can be generalized by shifting and rotating the cone.

2.5. Reward Function Design

Geometrically, to improve the observability of AON, orbital maneuvers must change the LOS at less an observation angle $\theta$. In this study, the observability criteria in [45] are adopted. At time $t_k$, when the spacecraft state is ${\mathit{x}}_k$, the linear independence between the relative position vector ${\displaystyle {\overline{\mathit{x}}}_{k+1}^p}={\mathrm{\Phi }}_r{\mathit{x}}_k$ obtained without an orbit maneuver and the relative position vector ${\mathit{x}}_{k+1}^p={\mathrm{\Phi }}_r{\mathit{x}}_k+G_r{\mathit{u}}_k$ obtained with an orbit maneuver should be maximized. Therefore, the dot product $J_{o,k}$ of ${\displaystyle {\overline{\mathit{x}}}_{k+1}^p}$ and ${\mathit{x}}_{k+1}^p$ should be minimized and

$$J_{o,k}={\left({\mathrm{\Phi }}_r{\mathit{x}}_k\right)}^T\left({\mathrm{\Phi }}_r{\mathit{x}}_k+G_{\mathit{r}}{\mathit{u}}_k\right)=\left|{\mathrm{\Phi }}_r{\mathit{x}}_k\right|\cdot \left|{\mathrm{\Phi }}_r{\mathit{x}}_k+G_r{\mathit{u}}_k\right|\cdot \cos \theta$$

(18)

Since the spacecraft should not fly past the target after the orbital maneuver, Equation (18) should also obey

$$J_{o,k}>0$$

(19)

In addition, it is also necessary to meet the constraints of position and velocity of the terminal interception, maximum thrust amplitude, and visibility cone angle and punish any violated constraint.

Multiobjective optimization seeks a balanced compromise between multiple optimal objectives. Therefore, there is no unique optimal solution but rather a Pareto optimal solution set. The improvement of solutions in this solution set on one goal is at the cost of reducing another goal. A simple processing method designs the optimization objective as a weighted sum of multiple objectives. Therefore, the instantaneous reward $R_k$ obtained at time $t_k$ is

$$R_k=-\frac{\Vert {\mathit{u}}_k\Vert }{N\Delta v_{\max }}-{\lambda }_1\frac{C_{o,k}}{50N}-{\lambda }_2{e}_{a,k}-{\lambda }_3\max \left(0,e_{x,k+1}-\epsilon \right)-{\lambda }_4\max \left(0,e_{b,k+1}-\epsilon \right)$$

(20)

where

$$C_{o,k}=\{\begin{array}{cc}{J}_{o,k}& J_{o,k}>0\\ 10000& J_{\mathit{o},k}\le 0\end{array}$$

(21)

$$e_{a,k}=\max \left\{0,\Vert {\mathit{u}}_k\Vert /\Delta v_{\max }-1\right\}$$

(22)

$$e_{x,k}=\{\begin{array}{c}0\\ \max \left\{\Vert {\mathit{r}}_N-{\mathit{r}}_f\Vert /\overline{r},\Vert {\mathit{v}}_N-{\mathit{v}}_f\Vert /\overline{v}\right\}\end{array} \begin{array}{c}k<N\\ k=N\end{array}$$

(23)

$$e_{b,k}=\max \left\{0,\max \left\{B{\mathit{x}}_k\right\}/\overline{r}\right\}$$

(24)

where $e_{a,k}$ is the penalty for violating the maximum thrust amplitude constraint, $e_{x,k}$ is the penalty for violating the terminal constraints, and $e_{b,k}$ is the penalty for violating the visibility cone constraint. The penalty for $J_{o,k}\le 0$ is incorporated into the $C_{o,k}$. $\overline{r}=1 km$ and $\overline{v}=0.01 km/\mathit{s}$ are normalization factors. $\epsilon$ represents the tolerance for the violation of constraints and constraints should be tightened with training, so the penalty relaxation factor should be decreased as

$$\epsilon \left(t\right)=\{\begin{array}{c}5\times {10}^{-2}\\ 5\times {10}^{-4}\end{array} \begin{array}{c}t\le T/2\\ t>T/2\end{array}$$

(25)

${\lambda }_1,{\lambda }_2,{\lambda }_3,{\lambda }_4$ are the weight factors of the corresponding items in the reward function with typical values ${\lambda }_1=0.1,{\lambda }_2=100,{\lambda }_3=1,{\lambda }_4=1$. Note that the values of ${\lambda }_1,{\lambda }_2,{\lambda }_3$ and ${\lambda }_4$ were investigated through grid search and the typical values were chosen to present a general performance of the algorithm. It can be seen that the introduction of observability in the reward function will add a negative score.

2.6. RL Algorithm

The PPO algorithm is adopted [21]. The PPO algorithm is one of the most widely used strategy gradient algorithms in continuous control. It learns a policy network to directly map the state to an action rather than maximizing the global value function. Therefore, PPO is suitable for online rendezvous guidance problems. PPO uses a typical actor–critic structure, as shown in Figure 3.

Using the multilayer neural network structure in [4], the policy strategy network consists of an input layer with $n_i=7$ neurons, an output layer with $n_o=3$ neurons, and three hidden layers of size $h_1=10n_i,h_2=\sqrt{h_1{h}_3},h_3=10n_o$, as shown in Figure 4. The activation function of all hidden layers is the hyperbolic tangent. The critical network structure is the same as the policy network, except $n_o=1$ [23].

The actor is a policy network ${\pi }_{\theta }\left({\mathit{a}}_k|{\mathit{o}}_k\right)$, where $\theta$ is the parameters of the policy network. The actor generates a control action based on the current state. The critic uses a multilayer neural network to estimate the value function and evaluate the performance of the action. The value function is the sum of the expected value of the sum of instantaneous reward $R_k$ along the trajectory episode with discount factor $\gamma$ according to the strategy ${\pi }_{\theta }$, that is,

$$V_k^{{\pi }_{\theta }}\left(\mathit{x}\right)=\underset{\tau ~{\pi }_{\theta }}{𝔼}\left[{\displaystyle \sum _{i=k}^{N-1}{\gamma }^{i-k}{R}_i\left({\mathit{x}}_i,{\mathit{a}}_i\right)|{\mathit{x}}_i=\mathit{x}}\right]$$

(26)

Both the actor and the critic use a gradient method and small batches of samples to update network parameters $\theta$.

The loss of the value network of the original actor–critic network is calculated as

$$A_i^{{\pi }_{\theta }}={\displaystyle \sum _{i=k}^{N-1}{R}_i+\gamma V_{\theta }\left({\mathit{o}}_{i+1}\right)}-V_{\theta }\left({\mathit{o}}_i\right)$$

(27)

A factor of the general advantage estimator (GAE) $\lambda$ is added to obtain

$${\widehat{A}}_i^{{\pi }_{\theta }}={\displaystyle \sum _{i=k}^{N-1}{\left(\gamma \lambda \right)}^{i-k}\left(R_i+\gamma V_{\theta }\left({\mathit{o}}_{i+1}\right)-V_{\theta }\left({\mathit{o}}_i\right)\right)}$$

(28)

If the update step of the policy network is too large, which will lead to the nonconvergence of the network, the PPO algorithm adds a truncation term and obtains a truncation policy objective function $J^{clip}\left(\theta \right)$ as

$$J^{clip}\left(\theta \right)=\frac{1}{N}{\displaystyle \sum _{k=0}^{N-1}\min \left(p_k\left(\theta \right)A_k^{{\pi }_{{\theta }_{\mathit{o}ld}}},clip\left(p_k\left(\theta \right),1-\kappa ,1+\kappa \right)A_k^{{\pi }_{{\theta }_{\mathit{o}ld}}}\right)}$$

(29)

where the strategy probability ratio $p_k\left(\theta \right)$ is calculated as

$$p_k\left(\theta \right)=\frac{{\pi }_{\theta }\left({\mathit{a}}_k|{\mathit{o}}_k\right)}{{\pi }_{{\theta }_{\mathit{o}ld}}\left({\mathit{a}}_k|{\mathit{o}}_k\right)}$$

(30)

$$clip\left(p_k\left(\theta \right),1-\kappa ,1+\kappa \right)=\{\begin{array}{c}1-\kappa ,\\ 1+\kappa ,\\ p_k\left(\theta \right),\end{array} \begin{array}{c}{p}_k\left(\theta \right)<1-\kappa \\ p_k\left(\theta \right)>1+\kappa \\ 1-\kappa <p_k\left(\theta \right)<1+\kappa \end{array}$$

(31)

where $\kappa$ is a small constant used to limit the learning speed. The function $clip$ is constrained in the range of $\left(1-\kappa ,1+\kappa \right)$, so it indirectly limits the speed of policy update. Therefore, according to the above truncation strategy objective function, the strategy network parameters $\theta$ are updated according to the gradient direction

$${\theta }_{new}={\theta }_{\mathit{o}ld}+{\alpha }_{\theta }{\nabla }_{\theta }J\left(\theta \right)$$

(32)

where ${\alpha }_{\theta }$ is the learning rate.

The implementation details and hyper parameters are shown in

Table 1. Super parameters of the PPO algorithm.

Super Parameter	Symbol	Value
Discount factor	$\gamma$	0.9999
GAE factor	$\lambda$	0.99
Learning rate	${\alpha }_{\theta }$	2.5 × 10−4
Clip range	$\kappa$	0.3
Training step	$T$	200 × 106
Parallel workers	$n_p$	60
Episodes per batch	$n_b$	4
SGA epochs per update	$n_e$	30

, which mainly refer to Ref. [23].

3. Flexible SaG Strategy

In the well-known optimal control problem (OCP) of rendezvous guidance, the number of orbital maneuvers is usually fixed, which is usually designed according to the overall goal of the mission [52]. This means that the solution of OCP for rendezvous guidance is dependent on the number of orbital maneuvers. Once the number of maneuvers changes, the OCP needs to be solved again. This inflexibility means additional computational burden and reduces real-time performance. Nevertheless, the close approach process is very dynamic, and the environment near the asteroid is highly unknown. We hope to overcome this inflexibility. To this end, we consider a SaG strategy.

The basic idea of the SaG strategy is to insert a certain number of SaG points into the original trajectory. At the SaG point, the spacecraft obtains the estimation of the current motion state according to the output of the navigation filter, and carries out the “stop” maneuver accordingly. The “stop” maneuver brakes the spacecraft’s velocity to zero, and the motion state just before the stop maneuver is saved. During the “stop” period, the navigation filter continues to work, and the additional “stop” maneuver is used to improve the range observability for AON. After “stopping” for a certain period of time, the spacecraft carries out the “go” maneuver to restore the spacecraft to the motion state before stopping. The navigation filter continues to work, and the additional “go” maneuver is used to improve the range observability again. Assuming that the “stop” time of the spacecraft is not long, other attributes of the mission will not be influenced. Then, the approach trajectory remains relatively unchanged in geometry after the SaG strategy is incorporated, but has a time delay. In this way, the SaG strategy can give some freedom to mission adaption.

Now we will formulate the SaG strategy. Let $t_s$ be the time of stop maneuver, ${\hat{\mathit{x}}}_{t_s}={\left[\begin{array}{cc}{\hat{\mathit{r}}}_{t_s}& {\hat{\mathit{v}}}_{t_s}\end{array}\right]}^T$ be the spacecraft state output by navigation filter at time $t_s$, where ${\hat{\mathit{r}}}_{t_s}$ is the estimated position vector and ${\hat{\mathit{v}}}_{t_s}$ is the estimated velocity vector. Then, the stop maneuver is $\Delta {\mathit{v}}_{\mathit{s}}=-{\hat{\mathit{v}}}_{t_s}$, and the time between stop maneuver and go maneuver is $\Delta t_{sg}$. The time of go maneuver is $t_s+\Delta t_{sg}$, and the go maneuver is $\Delta {\mathit{v}}_g={\hat{\mathit{v}}}_{t_s}$.

Nevertheless, due to navigation error and maneuver error, the spacecraft has a certain residual velocity after the “stop” maneuver, as shown in Figure 5. Therefore, during the “stop” period, the residual motion of the spacecraft will cause a certain drift, as shown in Figure 5. The drift during the “stop” period will cause the subsequent trajectory to deviate from the original trajectory, and the deviation will continue to diverge through error propagation, as shown in Figure 6. It will eventually lead to a large deviation from the object terminal state. The drift is actually a challenge to the robustness of rendezvous trajectory or rendezvous guidance. It is hard for the deterministic method for solving the OCP of rendezvous guidance to overcome this problem.

It is worth noting that the time delay and motion state drift caused by the SaG strategy can be well solved by a special design with the robust DRL-based guidance strategy. First, the input of the strategy network is a seven-dimensional vector composed of relative position, velocity, and left time. Therefore, it is only necessary to suspend the SaG strategy during the stop period, and then continue to execute the DRL-based guidance by inputting the output of the navigation filter into the strategy network, so that the whole process of navigation and guidance will be consistent. Second, as mentioned above, after the “stop” maneuver, the spacecraft does not actually stand still, but retains a small residual speed. Therefore, there will be a small residual motion during the SaG period, resulting in a certain spatial deviation between the starting state and the end state of the SaG period, as shown in Figure 6. Note that the spatial deviation will be small and bounded if there is a closed-loop station-keeping controller (such as the box control), general in asteroid missions [53], or the SaG period is small, and that the spatial deviation can be large if the station-keeping controller does not work and the SaG period is large. Nevertheless, the robustness of the DRL-based guidance will work in this situation. It can ensure the accuracy of terminal position and velocity state in the presence of initial motion state deviation. In this way, the SaG strategy can further improve the range observability. This also better explains the term “SaG point”, which means that in the framework of DRL-based robust guidance, the drift motion during the “stop” period can be ignored and can be regarded as a stationary point. We expect that when incorporating the SaG strategy, robust guidance will achieve better terminal accuracy than the open-loop guidance will do, as shown in Figure 7.

The number of SaG points is flexible. The SaG point can be triggered in different ways. For example, it can be triggered by the internal conditions of the navigation filter. For example, it can be triggered by the covariance matrix of the navigation filter to help accelerate the convergence. In addition, the SaG point can be triggered by an external trigger signal. Under the framework of online guidance, we propose an externally triggered SaG strategy, as shown in Figure 8. Figure 9 shows the online guidance framework obtained from AON and DRL-based guidance combined with the SaG strategy.

4. Numerical Simulation

4.1. Simulation Conditions

Taking 2016HO3 as the object asteroid (see

Table 2. Orbit elements of 2016HO3.

Epoch 2459600.5 (21 January 2022.0) TDB Reference: Heliocentric J2000 Ecliptic
Element	Value	Units
Semimajor axis	1.001137344063433	AU
Eccentricity	0.1029843787386461
Inclination	7.788928644671124	deg
Longitude of the ascending node	66.0142959682462	deg
Perihelion argument angle	305.6646720090911	deg
Mean anomaly	107.172338605596	deg

for its orbital elements), a simulation study of DRL-based guidance with the SaG strategy for the close approach was carried out. Note that although the mass of 2016HO3 is not small enough to ignore its gravity, we only use its orbit elements to demonstrate the proposed algorithm. In a typical R-bar approach scenario, it is planned to approach from ${\mathit{r}}_0=\left[\begin{array}{ccc}-10,000& 0& 0\end{array}\right]m$, ${\mathit{v}}_0=\left[\begin{array}{ccc}0& 0& 0\end{array}\right]m/\mathit{s}$ to ${\mathit{r}}_f=\left[\begin{array}{ccc}-1,000/\sqrt{2}& -1,000/\sqrt{2}& 0\end{array}\right]km$, ${\mathit{v}}_f=\left[\begin{array}{ccc}0& 0& 0\end{array}\right]km/\mathit{s}$ by 3 impulse maneuvers. The approach must be in the visibility cone with a cone half-angle $\beta =45\mathrm{deg}$. The nominal total rendezvous time is 3 days. The SaG starts after 12 h and lasts for 3 h. The AON was used in this scenario.

Table 3. Simulation parameters.

Parameter	Value	Unit
Environment
Equivalent disturbance, 1σ	1 × 10−7	m/s2
Measurement Noise, 1σ	1	mrad
Sample time	10	s
Rendezvous Mission
Initial condition, ${\mathit{x}}_0=[x,y,z,\dot{x},\dot{y},\dot{z}]$	$[-10000,0,0,0,0,0]$	m m m
		m/s m/s m/s
Target condition, ${\mathit{x}}_f=[x,y,z,\dot{x},\dot{y},\dot{z}]$	$[-1000/\sqrt{2},1000/\sqrt{2},0,0,0,0]$	m m m
		m/s m/s m/s
Number of pulses, N	3
Total time, T	3	day
Planned maneuver interval time, $\tau$	12	hour
Start time of SaG	12	hour
Duration of SaG	3	hour
Navigation Initiation
Initial navigation error, 1σ	$[1000,1000,1000,1,1,1]$	m m m
		mm/s mm/s mm/s

summarizes the simulation parameters used. To verify the effectiveness of the proposed method, Monte Carlo simulation was conducted by introducing a specific level of perturbations. Each group of simulations consists of 200 realizations, and the statistics were analyzed.

The guidance and control strategy trained using DRL and SaG was evaluated through closed-loop simulation, which considers the uncertainty of the state navigation observation. In particular, the observation uncertainty of the navigation state was set to 1% of the relative range after navigation convergence. The RL model was trained under environments with no perturbation and with navigation errors. Correspondingly, the resulting control strategies are named ${\pi }^{\det erm},{\pi }^{robust}$. The parameters for different uncertainty models corresponding to different control strategies are shown in

Table 4. Different control strategies and different environments.

Policy Name	Environment Name	Observation Uncertainty
${\pi }^{\det erm}$	$En{v}^{unp}$	0
		0
${\pi }^{robust}$	$En{v}^{obs}$	1% range
		5 mm/s

.

The neural network was implemented using the Python and TensorFlow framework [54]. The PPO was implemented by stable baselines [55], an open-source library containing a set of improved implementations of RL algorithms. The network training using RL took approximately 26 h with 60 parallel workers on a computer equipped with two 10-core Intel Xeon Silver 4210R CPUs @2.40 GHz. It is worth noting that once the training is completed, the computational cost of closed-loop control simulation verification of the obtained control strategy can be ignored because only a few more matrix operations than the general relative orbit dynamics propagation remain.

4.2. AON Performance Improvement by SaG Strategy

First, we tested the potential of the SaG strategy to improve AON performance. The performance of AON is mainly represented by the convergence speed and the navigation accuracy. We conducted Monte Carlo simulation to compare AON performance before and after adding the SaG strategy. Monte Carlo simulation includes 200 implementations with randomly generated perturbations and noises (see Table 3), and statistical values are taken for analysis.

Figure 10 shows that after adding the SaG point, the convergence speed is greatly accelerated, and the convergence time is advanced from the original second maneuver time to the stop maneuver time. In addition, after the SaG point is added, the navigation accuracy after navigation convergence is also improved. Specifically, the terminal navigation position error is reduced from 76.5 m to 2 m, and from the >5% range to the <1% range, as shown in Figure 11. Figure 10 and Figure 11 also show that the navigation error drops rapidly during the SaG period. It is satisfactory for the rendezvous guidance to have the navigation error decrease to less than 1% as soon as possible. After adding the SaG point, the navigation error decreases to less than 1% after 0.3 $t_f$, while the navigation error of the original guidance is always >1% range. Figure 12 shows the AON velocity error with and without SaG. It is seen that the AON velocity error without SaG never converges, but the AON velocity error with SaG converges after the SaG period. Specifically, the terminal navigation velocity error is reduced from 3 mm/s to <0.07 mm/s.

4.3. Deterministic Guidance Combined with SaG Strategy

Deterministic guidance combined with the SaG strategy was simulated using the online guidance framework, as shown in Figure 9. The closed-loop simulation results of deterministic guidance with the SaG strategy are shown in Figure 13. The blue line represents the nominal trajectory of the deterministic policy ${\pi }^{\det erm}$ running on the unperturbed environment $En{v}^{unp}$. The gray lines represent the Monte Carlo sample trajectories of ${\pi }^{\det erm}$ running on the observation uncertain environment $En{v}^{obs}$. It can be seen that the nominal trajectory is a dog-leg one that can improve the range observability of AON. The characteristic of the nominal trajectory is shown in

Table 5. The characteristic of the nominal trajectory for ${\pi }^{\det erm}$.

Time Step	$\mathbf{Maneuver} \mathbf{\Delta }\mathit{V},\mathit{m}/\mathit{s}$	$\mathbf{Observation} \mathbf{Angle} \mathit{\theta },\mathbf{deg}$
0	0.022	18.35
1	0.015	18.48
2	0.017	121.21
3	0.023	-

. It can be seen that the sample trajectories of ${\pi }^{\det erm}$ running on $En{v}^{obs}$ have a certain spatial deviation from the trajectory of ${\pi }^{\det erm}$ running on $En{v}^{unp}$, especially at the SaG point, the deviation increases significantly. Specifically, the position deviation of the trajectory at the SaG point increased from 10 m to 1000 m, and the speed deviation increased from 1 mm/s to 5 mm/s. This is caused by the state estimation error of the navigation filter, which is also consistent with our expectations in Section 3 (see Figure 8). Subsequently, the deviation increases continuously through propagation, which greatly increases the terminal error of close approach rendezvous, as shown in Figure 14. Specifically, the terminal position error increased to 977.2 m.

This cannot meet the task requirements. This shows that although the SaG strategy can improve the AON performance, the state drift introduced by the residual speed during the “stop” period will erode the original deterministic guidance.

4.4. Robust Guidance Combined with SaG Strategy

Robust guidance combined with the SaG strategy was simulated also using the online guidance framework, as shown in Figure 9. The closed-loop simulation results of robust guidance with the SaG strategy are shown in Figure 15.

The blue line represents the nominal trajectory of the deterministic policy ${\pi }^{robust}$ running on the unperturbed environment $En{v}^{unp}$. The gray lines represent the Monte Carlo sample trajectories of ${\pi }^{robust}$ running on the observation uncertain environment $En{v}^{obs}$. It can be seen that the nominal trajectory is a dog-leg one that can improve the range observability of AON. The characteristic of the nominal trajectory is shown in

Table 6. The characteristic of the nominal trajectory for ${\pi }^{\mathit{r}\mathit{o}b\mathit{u}\mathit{s}t}$.

Time Step	$\mathbf{Maneuver} \mathbf{\Delta }\mathit{V},\mathit{m}/\mathit{s}$	$\mathbf{Observation} \mathbf{Angle} \mathit{\theta },\mathbf{deg}$
0	0.026	25.78
1	0.027	21.67
2	0.015	138.12
3	0.024	-

. It can be seen that the sample trajectories of ${\pi }^{\det erm}$ running on $En{v}^{obs}$ have a certain spatial deviation from the trajectory of ${\pi }^{\det erm}$ running on $En{v}^{unp}$, especially at the SaG point, the deviation increases significantly. Specifically, the position deviation of the trajectory at the SaG point increased from 10 m to 1000 m, and the speed deviation increased from 1 mm/s to 5 mm/s.

This is consistent with our expectations in Section 3 (see Figure 8). However, the subsequent development is quite different from Figure 13. The deviation decreases continuously through propagation, which greatly reduces the terminal error of close approach rendezvous, as shown in Figure 16. Specifically, the terminal position error was reduced to 3.578 m, as shown in

Table 7. Terminal accuracy of different policies.

Policy Name	Environment	$\mathbf{Fuel} \mathbf{Cost} \mathbf{\Delta }\mathit{V},\mathit{m}/\mathit{s}$		$\mathbf{Terminal} \mathbf{Errors} \mathbf{\Delta }\mathit{r},\mathit{m}$
		Mean	Std	Mean	Std
${\pi }^{\det e\mathit{r}m}$	$En{\mathit{v}}^{\mathit{u}np}$	0.07780	-	11.51	-
	$En{\mathit{v}}^{\mathit{o}b\mathit{s}}$	0.0825	0.0072	977.2	406.8
${\pi }^{\mathit{r}\mathit{o}b\mathit{u}\mathit{s}t}$	$En{\mathit{v}}^{\mathit{u}np}$	0.09175	-	2.123	-
	$En{\mathit{v}}^{\mathit{o}b\mathit{s}}$	0.0943	0.0046	3.578	3.436

.

Therefore, the robust guidance gradually absorbs the large trajectory deviation caused by the drift of the SaG point. This makes the terminal error of close approach rendezvous remain at a small level. This shows that the robust guidance strategy can be well compatible with the SaG strategy. It can not only improve the AON performance, but also absorb the state drift introduced by the residual speed during the “stop” period.

Through the combination of the SaG strategy and the robust guidance strategy based on DRL, the spacecraft can stop at any time in the process of approaching the asteroid, which gives the approach process more mission flexibility because stop time can be used for detection and debugging. The guidance need not be solved again, so the computational complexity is reduced. Because the onboard computing resources are very limited, this is very beneficial.

5. Conclusions

Aiming at the need for more adaptability and robustness during the asteroid close approach phase, a flexible and robust guidance framework based on DRL and SaG strategies is proposed in this paper. A good match between the SaG strategy and the DRL-based robust guidance is demonstrated. The SaG strategy will introduce a state drift, but the DRL-based guidance will absorb the state deviation to ensure a better terminal accuracy. The proposed framework has the ability to increase the number of orbital maneuvers while basically maintaining the geometry of the original trajectory. Such an ability to make temporary braking and then restart freely in the original rendezvous trajectory is very attractive. It makes the autonomous mission planning of the rendezvous process easier to realize. More accurate dynamics, including the gravity of the asteroid and the SRP, will be considered in future work.