Cross-Spectral Navigation with Sensor Handover for Enhanced Proximity Operations with Uncooperative Space Objects

Abstract

Close-proximity operations play a crucial role in emerging mission concepts, such as Active Debris Removal or small celestial bodies exploration. When approaching a non-cooperative target, the increased risk of collisions and reduced reliance on ground intervention necessitate autonomous on-board relative pose (position and attitude) estimation. Although navigation strategies relying on monocular cameras which operate in the visible (VIS) spectrum have been extensively studied and tested in flight for navigation applications, their accuracy is heavily related to the target’s illumination conditions, thus limiting their applicability range. The novelty of the paper is the introduction of a thermal-infrared (TIR) camera to complement the VIS one to mitigate the aforementioned issues. The primary goal of this work is to evaluate the enhancement in navigation accuracy and robustness by performing VIS-TIR data fusion within an Extended Kalman Filter (EKF) and to assess the performance of such navigation strategy in challenging illumination scenarios. The proposed navigation architecture is tightly coupled, leveraging correspondences between a known uncooperative target and feature points extracted from multispectral images. Furthermore, handover from one camera to the other is introduced to enable seamlessly operations across both spectra while prioritizing the most significant measurement sources. The pipeline is tested on Tango spacecraft synthetically generated VIS and TIR images. A performance assessment is carried out through numerical simulations considering different illumination conditions. Our results demonstrate that a combined VIS-TIR navigation strategy effectively enhances operational robustness and flexibility compared to traditional VIS-only navigation chains.

1. Introduction

In recent years, researchers have dedicated considerable attention towards operations in close-proximity to uncooperative orbiting artificial objects. Within this context, the onboard reconstruction of the chaser-target relative state vector is a crucial capability for incoming mission scenarios such as formation flying missions (FF), on-orbit servicing demonstrators (OOS), and active debris removal, as well as small bodies exploration [1]. While these missions are currently in the spotlight of discussions, attaining feasibility still hinges on substantial technological advancements. The necessity for close-proximity manoeuvring introduces a requirement for a guidance, navigation, and control chain to be autonomously solved onboard to ensure timeliness, reactivity, effectiveness, and robustness in both nominal and off-nominal operations. The initial component of this chain is the relative state reconstruction and navigation. This is especially challenging when dealing with artificial uncooperative targets, which calls for a robust solution relying solely on the chaser’s capabilities, as highlighted in [2,3]. In this operational context, imaging with passive sensors emerges as the best sensor architecture. A comprehensive review of initial pose determination techniques based on a VIS monocular camera is provided in [4]. Further, VIS-based optical navigation has been successfully applied within both cooperative and uncooperative rendezvous, as pointed out in [5,6]. Nonetheless, the effectiveness of visible imaging is heavily reliant on illumination conditions. Consequently, OOS missions encounter substantial constraints when illumination requirements for VIS image acquisition are integrated into the design and definition of the close proximity operations. Elements such as the target orbit beta angle, attitude history, solar aspect angle induced by the chaser’s fly-around, and the camera axis could significantly jeopardize the ability to detect and track the target appropriately. This limitation may lead to an unacceptable increase in either mission duration or risk. Illumination bottlenecks become particularly pronounced for targets in Low Earth Orbit (LEO) experiencing prolonged eclipses, as highlighted in [7]. In recent years, the Hayabusa 2 mission successfully exploited its thermal-infrared (TIR) imager for vision-based navigation purposes [8]. Such an outcome has highlighted the possibility of combining sensors operating in different spectral bands to enhance the navigation solution accuracy. This work aims at exploiting a TIR imager leveraging its insensitivity to illumination conditions, to overcome the limitation imposed by imaging sensors operating in the visible spectrum. Some preliminary work on the topic can be found in [9], in which the author performs an assessment of the best feature detector and descriptors for thermal infrared images. Nevertheless, TIR sensors are usually characterized by a smaller array size compared to visible ones, thus they have a lower resolution and poorer contrast with respect to VIS sensors, which negatively affects image processing algorithms, as highlighted in [10]. To overcome these limitation, sensor fusion strategies can play a major role, as pointed out in [11]. Multispectral data fusion is a dominant technique within the field of Unmanned Aerial Vehicle (UAV) navigation [12], yet its role remains marginal in the domain of spacecraft relative navigation. The different multispectral data fusion schemes can be divided into two main approaches: image fusion and high-level data fusion, as portrayed in Figure 1.

Multispectral image fusion aims at creating a new and more informative image type by combining the complementary strengths of the two distinguished spectral bands. The newly obtained image type can then be fed to the subsequent navigation chain to enhanche its robustness to illumination conditions. Image fusion has been successfully applied within the context of remote imaging [13], while its application in spacecraft navigation scenarios remain marginal. The work presented in [14] is concerned with the evaluation of the different pixel-level image fusion techniques within the context of spacecraft relative navigation. The validity of image fusion techniques for navigation purposes is assessed in [15,16]; where the authors test the effectiveness of pose initialization algorithms on VIS-TIR fused images. A further step is then presented in [17], in which a Convolutional Neural Network (CNN) based pose estimation algorithm is successfully tested on this new image type. Despite the effectiveness of pixel-level image fusion, this work focuses on high level data fusion. Decision-level multispectral data fusion allows for more flexibility and robustness of the whole navigation chain. In fact, the two source images are processed separately and they are treated as two independent sensors. In this way, the information channels can be treated as redundant, and thus it is easier to isolate faulty measurements or to exclude one sensor when it is no longer providing meaningful information. The work presented in [18] presents a navigation architecture in which feature tracking is performed simultaneously on both VIS and TIR images, and this information is then fused within a Kalman Filter to achieve robustness in person motion tracking. Subsequently, the idea of performing high-level multispectral data fusion was adopted for relative navigation and mapping of asteroids and unknown spacecraft by [19,20], respectively. Further research on asteroid relative navigation is carried out in [21], in which the authors perform a CNN-based feature map fusion to enhance the accuracy of centroid detection during proximity operations of the HERA mission. In the presented research, the author build on the approach presented in [22], to propose a flexible navigation strategy to fuse multispectral information. Specifically, the aforementioned work introduces the concept of sensor handover from onecamera to the other to support night-time navigation for a mobile robot. The results presented in the paper suggest that the navigation accuracy and robustness can clearly benefit from the introduction of a thermal infrared imager. However, the handover from the VIS to the TIR camera is controlled by an external user; whereas we implement a fully automatic switch between the two different sensing modalities. Feature detection and feature tracking are performed separately for VIS and TIR images, and the feature points position is fed as observable to an Extended Kalman Filter (EKF). Model-to-image matching is then employed to establish 3D-2D correspondences between the known model and the extracted feature points. Such tightly coupled architecture can be affected in terms of robustness whenever the target’s shape is particularly complex, since it may be more challenging to accurately detect and track a high number of features. However, given the relatively simple shape of the Tango spacecraft and the utilization of two cameras simultaneously, these robustness issues are easily mitigated. Further, we introduce autonomous sensor handover from one sensing modality to the other, in such a way to retain only the most meaningful measurements source. This approach may be extremely useful during eclipses, where the VIS camera is automatically excluded from the navigation chain, since it cannot contribute to the pose estimation task. The presented navigation scheme is applicable to known uncooperative targets, for which at least a simplified geometrical model is available. It is important to remark that the target geometry of an uncooperative space object may not be always known. The most common way of dealing with unknown uncooperative objects is to rely on Simultaneous Localization and Mapping (SLAM) techniques, which enable the chaser to both reconstruct the target’s shape and to perform relative navigation. However, such techniques tend to be numerically heavy, due to the high number of map points which shall be stored in the memory. As proposed in [23], it can be useful to split the mission into two operative phases: the first one relies on SLAM to gather information about the target’s shape; while in the second phase of the mission the known geometry of the target object is exploited to perform model-based relative navigation. Building on this concept, we can assume to be beyond the preliminary mapping phase, and that at least partial information regarding the target’s shape can be exploited. The major contributions of the paper can be then summarized as follows:

• Development of a tightly-coupled navigation chain for cross-spectral relative navigation, capable of performing autonomous sensor handover among the different sensing modalities.
• Quantitative assessment of the advantages of introducing a TIR imager in the navigation chain through numerical simulations and high-fidelity image rendering tools.

The paper is organized as follows: Section 2 presents a schematic outline of the developed navigation architecture, highlighting the functionalities of each building block. The Image Processing (IP) functional block is thoroughly described in Section 3; while Section 4 details the relative navigation filter. The simulation environment is introduced in Section 5, while the results are presented and discussed in Section 6. Conclusion summarizing the main outcomes of our study are reported in Section 7.

2. Navigation Chain

A block diagram of the proposed visual navigation architecture is shown in Figure 2. The navigation chain can operate either using VIS or TIR images, or both of them. The image processing functional block performs feature detection and tracking; while an Extended Kalman Filter (EKF) estimates the relative chaser-target pose. The filter employs the target point-features positions on the images as measurements and estimates the relative pose by comparing them with the expected position of the matching target’s model landmarks projection on the image plane. Our EKF exploits a target model which is built offline on the target’s geometry information, as similarly done in [2,24].

3. Image Processing

The image processing functional block is in charge of extracting features and tracking them across the incoming images. Periodically, feature re-initialization is performed. Lastly, sensor handover is performed from one camera to the other when necessary.

3.1. Features Detection and Tracking

Among different feature detectors, Oriented Fast and Rotated Brisk (ORB) [25] has been selected for this work to be applied both to VIS and TIR images; due to its robustness to challenging illumination conditions and scale variations. ORB feature detection has been used in a wide number of application for relative spacecraft navigation in the visible spectral band [26]. On the other hand, the literature concerning the evaluation of feature detection algorithms on TIR images is scarce. A preliminary performance evaluation of different detector and descriptor is available in [27], in which different feature detector and descriptors are tested on synthetic thermal infrared images. This work shows that ORB features offer a good compromise between detection accuracy and computational time, and that they are well-suited for feature tracking. The number of features is restrained to 250 to reduce the computational burden. The detected feature points are then tracked across the subsequent images using Lucas-Kanade tracking algorithm [28].

3.2. Features Re-Initialization

The number of tracked features decreases throughout the sequence images due to the relative motion between the chaser and the target. To keep the number of tracked features high enough to ensure a reliable pose estimation, a dedicated routine is implemented to detect and match new features to be added to the already tracked set. This process is initiated when the ratio between the convex hull of the tracked image features and the convex hull formed by projecting all the landmarks onto the images falls below a predetermined value. This indicates that only a portion of the target is currently encompassed by the tracked features. When this condition is verified new features are detected in the part of the image outside the convex hull of the tracked set. The newly identified candidate features are subsequently projected based on the homography that establishes the relationship between the tracked image features and the model’s landmarks. This enables the matching of these candidates to the model landmarks, through a straightforward nearest neighbour association. However, this method relies on correlations between previous features and landmarks to match new candidates. This dependence can introduce error propagation, especially if the initial matching is erroneous. To mitigate this issue, the navigation chain undergoes periodic full re-initialization of features. This process involves discarding the entire tracked set, detecting new features across the entire image, and matching them with the model landmarks. Without the possibility to use any existing features-to-landmarks correlations, it is necessary to perform 2D-3D point registration. A pseudocode of the algorithm designed to re-initialize the features is reported in Algorithm 1. Initially, the algorithm calculates the convex hull encompassing all detected features and all model points projected according to the best state estimate, as shown in Figure 3a. Assuming that estimation errors are constrained during the pose tracking process, the algorithm randomly guesses the association between the feature set’s peripheral points and the closest model landmarks’ peripheral points, which is shown in Figure 3b. The homography associated with these pairings is then computed and applied to all model landmarks. After this homographic transformation, it is reasonable to consider that the model landmarks and their respective image features are closely overlapped. Therefore, the matching of features to landmarks is computed using a nearest-neighbor association technique, as shown in Figure 3c. The peripheral points association, homographic transformation, and nearest-neighbor matching process are executed iteratively. Utilizing a RANSAC-like approach, the iteration that yields the most matched features is saved as the output of the routine. While this algorithm effectively tackles the 3D-2D registration problem with efficiency, it exhibits decreased robustness when faced with relative attitude errors exceeding 15 degrees. This implies its suitability primarily for the re-initialization of features during the pose tracking process. The feature reinitialization procedure is summarized in Algorithm 1.

Algorithm 1 Feature Re-initialization

1: Given I: set of image feature locations on the image plane 2: Given M: set of model landmark locations on the image plane 3: Given $n_{max}$: maximum number of feature points 4: Given $i_{\max }$: maximum number of iterations 5: procedure Re-initialize Features($I,M,n_{max}$) 6:     Identify boundary point sets $I_b$ and $M_b$ from sets I and M 7:     for $i\leftarrow 1$ to $i_{\max }$ do 8:           Randomly pair four coordinates from set $I_b$ with four from set $M_b$ 9:           Compute homography H mapping these pairs 10:         Project set M onto set I using homography H 11:         Identify the matching pairs between I and M using nearest neighbor association 12:         if number of matched pairs $>n_{\max }$ then 13:            result ← matched pairs 14:            $n_{\max }$← number of matched pairs 15:         end if 16:     end for 17:     Return result 18: end procedure

3.3. Sensing Modalities Switch

For optimal performance, the navigation chain is designed to leverage both spectral information whenever possible. Conversely, a sensing modality switch is introduced to automatically discard one of the two spectra when it lacks informative content. Concerning thermal images, they are excluded during the re-initialization process if reliable results are not obtained (i.e., the number of features output by the routine is lower than the trigger for re-initialization). The same is done for visible images, but in addition a further check on the pixel intensity is performed. The visible images are used only if the number of pixels with an intensity above a threshold set at three times the standard deviation ($3\sigma$) of the image’s Gaussian noise exceeds a user-defined value. This value was calibrated by considering images useful if at least 10% of the region of interest, where the target is located, is illuminated. This additional condition is introduced because, in low-light conditions, the target loses clarity in visible images, while in thermal images, pixel intensity is preserved even under dynamic illumination conditions. If the switch discards a spectrum, periodic re-initialization tests occur to assess whether positive results can be obtained. If successful, the previously discarded spectrum may be reintroduced into the sensor fusion process, enhancing adaptability to changing environmental conditions. The spectra selection procedure is summarized in Algorithm 2.

Algorithm 2 Spectra Selection

1: Given $VIS$: current VIS image 2: Given $x_{VIS}$: VIS feature points 3: Given $TIR$: current TIR image 4: Given $x_{TIR}$: TIR feature points 5: Given $px\_intensity\_threshold$: pixel intensity threshold 6: Given $px\_number\_treshold$: number of activated pixels treshold 7: Given $feature\_threshold$: feature points threshold 8: procedure Select_spectrum($VIS,TIR$) 9:       $flag\_VIS\leftarrow True$ 10:     $flag\_TIR\leftarrow True$ 11:     if $sum(n\_pixels(VIS)>px\_intensity\_threshold)>px\_number\_treshold$ then 12:         $x_{VIS}\leftarrow Re\_initialize\_Features$ 13:         if $length\left(x_{VIS}\right)<feature\_threshold$ then 14:            $flag\_VIS\leftarrow False$ 15:         end if 16:     else 17:         $flag\_VIS\leftarrow False$ 18:     end if 19:     $x_{TIR}\leftarrow Re\_initialize\_Features$ 20:     if $length\left(x_{TIR}\right)<feature\_threshold$ then 21:         $flag\_TIR\leftarrow False$ 22:     end if 23:     Return $x_{VIS}$, $x_{TIR}$, $flag\_VIS$, $flag\_TIR$ 24: end procedure

4. Navigation Filter

We employ an EKF for relative pose estimation. The most common estimation techniques, i.e., EKF, Unscented Kalman Filter (UKF) and Particle Filter (PF) were taken into account [29]. The main driver for choosing the EKF is its computational cost. Adopting a linear model to describe the relative translational motion allows us to directly have an analytical expression for the translational part of the Jacobian matrix used to evaluate the State Transition Matrix (STM). This solution is extremely effective in terms of computational resources, thus driving our choice to the EKF rather than more complex filters. The filter is designed to estimate the relative position, velocity, attitude and angular rates of the target with respect to the chaser spacecraft. Since the observation model depends on both the relative position and attitude, the navigation filter needs to be coupled. The state vector is defined as:

$$\mathit{x}={\left[\begin{array}{cccc}{\mathit{\varrho }}^T,& {\dot{\mathit{\varrho }}}^T,& {\mathit{q}}^T,& {\mathit{\omega }}^T\end{array}\right]}^T$$

(1)

being $\mathit{\varrho }$ and $\dot{\mathit{\varrho }}$ the relative position and velocity between the two spacecraft’s centers of mass, $\mathit{q}$ the relative quaternion and $\mathit{\omega }$ the relative angular velocity.

4.1. Coordinate Systems

The reference frames considered in this paper include the Earth-Centered Inertial (ECI) coordinate system, represented by $\mathcal{I}$; a chaser-fixed Local Vertical, Local Horizon (LVLH) frame, denoted as $\mathcal{L}$, where the x-component aligns with the spacecraft’s radial direction, the z-component aligns with the orbit’s angular momentum, and the y-axis completes the right-hand triad; and two body-fixed reference frames, aligned with the target’s and chaser’s principal inertia axes, designated as $\mathcal{C}$ and $\mathcal{T}$ respectively. Without loss of generality, the chaser body frame $\mathcal{C}$ is assumed to coincide with the imaging sensors’ reference frame. Please notice that the LVLH frame is here defined on the chaser to exploit the knowledge of the chaser spacecraft true orbital motion, which is reflected in the mean motion parameter n.

4.2. Dynamical Model

The translational dynamical model selected for the propagation of the relative position is based on the Clohessy–Wiltshire [30] equations of unperturbed relative motion, which are reported in Equation (2).

$$\begin{array}{cc}\hfill \ddot{x}-2n\dot{y}-3n^{2}x& =0\hfill \\ \hfill \ddot{y}+2n\dot{x}& =0\hfill \\ \hfill \ddot{z}+n^{2}z& =0\hfill \end{array}$$

(2)

where, $\mathit{\varrho }={[x,y,z]}_{\mathcal{L}}$ represents the position of the target in the chaser’s LVLH frame, and n is the chaser’s mean motion. The attitude parametrization follows the formulation of a Multiplicative Extended Kalman Filter (MEKF) [31]. The filter propagates a three-element local attitude error $\mathit{a}$ formalized in Modified Rodrigues Parameters (MRP) while keeping track of a reference quaternion. For the propagation of the attitude error, the differential equation detailed in [32] from quaternion kinematics is used, as presented in Equation (3).

$$\dot{\mathit{a}}=-{\displaystyle \frac{1}{2}}[\mathit{\omega }\times ]\mathit{a}+\mathit{\omega }$$

(3)

The relative angular velocity vector $\mathit{\omega }$ between the target and the chaser, expressed in the chaser body frame, is defined as in Equation (4). Here, ${\mathit{\omega }}_{\mathcal{T}}$ and ${\mathit{\omega }}_{\mathcal{C}}$ represent the angular rates of the target and chaser in their respective rotating frames, and ${\mathit{A}}_{\mathcal{C}/\mathcal{T}}$ is the rotation matrix mapping the target body frame into the chaser body frame.

$${\left[\mathit{\omega }\right]}_{\mathcal{C}}={\mathit{A}}_{\mathcal{C}/\mathcal{T}}{\left[{\mathit{\omega }}_{\mathcal{T}}\right]}_{\mathcal{T}}-{\left[{\mathit{\omega }}_{\mathcal{C}}\right]}_{\mathcal{C}}$$

(4)

Equation (5) describes the dynamics of the relative angular rates, in which the inertia tensors of the chaser and target are labeled ${\mathbb{I}}_{\mathcal{C}}$ and ${\mathbb{I}}_{\mathcal{T}}$, ${\mathit{u}}_{\mathcal{C}}$ is the control torque of the chaser spacecraft and the subscript $\mathcal{C}$ has been omitted for brevity.

$$\begin{array}{cc}\hfill \dot{\mathit{\omega }}=& {\mathbb{I}}_{\mathcal{C}}^{-1}\{{\mathbb{I}}_{\mathcal{C}}{\mathit{A}}_{\mathcal{C}/\mathcal{T}}{\mathbb{I}}_{\mathcal{T}}^{-1}\left[-{\mathit{A}}_{\mathcal{C}/\mathcal{T}}(\mathit{\omega }+{\mathit{\omega }}_{\mathcal{C}})\times {\mathbb{I}}_{\mathcal{T}}{\mathit{A}}_{\mathcal{C}/\mathcal{T}}^T(\mathit{\omega }+{\mathit{\omega }}_{\mathcal{C}})\right]\hfill \\ & -\left[{\mathit{u}}_{\mathcal{C}}-{\mathit{\omega }}_{\mathcal{C}}\times {\mathbb{I}}_{\mathcal{C}}{\mathit{\omega }}_{\mathcal{C}}\right]-{\mathbb{I}}_{\mathcal{C}}({\mathit{\omega }}_{\mathcal{C}}\times \mathit{\omega })\}\hfill \end{array}$$

(5)

The filter performs a discrete propagation of the states and the associated covariance by computing the State Transition Matrix (STM) of the ordinary differential equations reported in Equations (2), (3) and (5).

4.3. Observation Model

The filter processes the projected positions of the target’s model landmarks onto the image plane, according to Equation (6)

$${\mathit{p}}_i=K({\mathit{A}}_{\mathcal{C}/\mathcal{T}}{\mathit{P}}_{\mathit{i}}+\mathit{\varrho })$$

(6)

In which ${\mathit{P}}_{\mathit{i}}$ is the i-th feature point position in the target’s reference frame and K is the intrinsic camera calibration matrix. A more detailed derivation of the observation model is available in [33] for the interested reader. The target’s landmarks are the vertices of a reduced CAD model of its shape. In the case of the target Tango, this model contains 170 vertices.

4.4. Measurement Noise Covariance Adaptation

For optimal functionality, the Extended Kalman Filter (EKF) necessitates precise modelling of the noise covariance of the measurements. This task holds particular significance in this research application, where a dynamic environment introduces time-varying measurement noise. To overcome these shortcomings, an online adaptation of the measurement noise covariance matrix based on the residual of the filter is implemented. At each iteration, the covariance matrix $\mathit{R}$ is updated according to the equation derived in [34] and reported in Equation (7).

$${\mathit{R}}_k=\alpha {\mathit{R}}_k+(1-\alpha )({\mathit{ϵ}}_k{\mathit{ϵ}}_k^T+{\mathit{H}}_k{\mathit{P}}_k^{+}{\mathit{H}}_k^T)$$

(7)

In Equation (7), $\mathit{ϵ}$ represents the filter’s residual, ${\mathit{H}}_k$ is the Jacobian of the measurement function, and ${\mathit{P}}_k^{+}$ is the updated covariance matrix of the states. A major advantage of this solution is that the noise covariance is adapted online for each target’s feature individually, progressively identifying and assigning less significance to the least reliable ones. The main parameter which rules the adaptation is the forgetting factor $\alpha$, which ranges from 0 to 1. A higher $\alpha$ value leads to a slower adaptation of the $R_k$ matrix, yet it provides more stability. In this work the value is set to $\alpha =0.8$; which represents a good compromise between adaptation capabilities and stability of the $R_k$ matrix.

4.5. Outliers Rejection

To increase the robustness of the filter an outlier rejection routine is introduced, as explained in [35] and successfully employed in [32]. The null-hypothesis test is performed, assuming that the measurement noise is Gaussian distributed, using the Mahalanobis distance as a figure of merit. By defining the filter’s innovation at the k-th step as ${\mathit{d}}_k$ and its covariance as ${\mathit{S}}_k$, the Mahalanobis distance ${\gamma }_k$ can be computed as shown in Equation (8).

$${\gamma }_k={\mathit{m}}_k^2={\mathit{d}}_k^T{\mathit{S}}_k^{-1}{\mathit{d}}_k$$

(8)

Under the assumption that the null hypothesis holds, ${\gamma }_k$ follows a Chi-square distribution. Therefore, all the measurements with a squared Mahalanobis distance exceeding a threshold k are excluded, defining k to ensure that Equation (9) holds.

$$P(k>\alpha )=0.05$$

(9)

5. Simulation Environment

5.1. Image Rendering

This work employs the open-source POV-Ray [36] to generate photorealistic spaceborne validated VIS images, as described in [37]. The VIS rendering tool used for this work has been validated against the SPEED dataset [38], which in turn has been validated against images of the PRISMA mission. It is worth noticing that the different illumination conditions can be easily simulated by the user by properly selecting the Sun position in the ECI frame, as well as the chaser and target absolute coordinates in the ECI frame. The Sun represents the only light source of the POV-Ray scene, and thus it is representative of a realistic scenario. Concerning TIR images, Blender 2.93 has been preferred as the main image rendering software, exploiting a tool internally developed by the ASTRA research group. The full description of the thermal infrared rendering tool is available in [39,40]. The tool exploits a validated thermal simulation to compute the temperature field of the Tango spacecraft. Such temperature field is then processed to obtain the heat flux captured by the camera, exploiting the relative pose between the camera and the target to compute the view factors. It is then possible to obtain the thermal image based on the assumption that the output Digital Number (DN) produced by the sensor is proportional to the heat flux received by the pixels. Furthermore, a thorough description of the characteristic noise of a thermal camera, that has been used for this work, is available in [41]. Please notice that the validation of TIR images is significantly more challenging due to the lack of TIR datasets. However, the model used for the thermal simulation of the Tango spacecraft has been validated against a thermovacuum experiment within the context of CubeSat testing and integration [42]. It is also acknowledged that the acquisition of real thermal infrared images requires a dedicated and calibrated facility, which is not currently available. The images produced using such tools are reported in Figure 4 where the VIS images (Left) are compared with the TIR images (Right) for a simplified Tango spacecraft model and the camera parameters reported in

Table 1. VIS and TIR camera parameters.

	Array Size [px]	FoV [deg]	Focal Length [mm]
VIS	1024 × 1024	[14, 14]	20
TIR	512 × 512	[14, 14]	20

.

For simplicity, the VIS and TIR camera centers are assumed to be coincident with the chaser spacecraft center of mass, and they are always pointing towards the target. Concerning the noise level of the synthetically generated images, VIS images are postprocessed by adding a white Gaussian noise with ${\sigma }^2$ = 0.0022 and blurred with a Gaussian blurring characterized by ${\sigma }^2$ = 1 and zero mean. The noise parameters have been selected equal to [43]. With regards to thermal imaging sensors, the research presented in [44] demonstrated that microbolometers are mostly affected by two sources of noise: the thermal noise and the $1/f$ noise. The former is a characteristic of all electronic devices and it is modeled as an additive white Gaussian noise, assuming the same characteristics adopted for VIS images. The $1/f$ noise, which is also referred to as flicker noise or pink noise, is instead dominant at low frequencies, as demonstrated by [45]. An additive pink noise can be numerically obtained by applying a suitably shaped low-pass filter to a white Gaussian noise. A two-dimensional Fourier transform is used to decompose white noise into the frequency domain. The amplitude (A) of each frequency is then scaled such that the higher the frequency, the lower the amplitude using the following relationship:

$$A^{\prime }=A{\displaystyle \frac{1}{{(f_x^2+f_y^2)}^{\alpha /2}}}$$

(10)

where $f_x$ and $f_y$ are the spatial frequencies and $\alpha$ is an exponent which determines the spectral slope ($\alpha =1$ for flicker noise). The inverse Fourier transform is then applied to convert the filtered result back to the spatial domain. The variance of the White Noise to be filtered is here assumed to be ${\sigma }^2=0.0022$. Similar to VIS images, also the TIR images are blurred with a Gaussian blurring characterized by $\sigma =1$ and zero mean.

5.2. Reference Dynamic

The reference trajectory is computed apriori. The translational motion is computed according to a perturbed restricted two-body model which accounts for the $J_2$ effect, that is the most dominant perturbation for satellites flying in LEO orbits. Furthermore, we consider the effects of solar radiation pressure and atmospheric drag. The former contribution is computed according to a simplified cannonball model, while the latter considers an exponential air model to describe the atmospheric density. The rotational motion is instead described through an unperturbed free motion, which is a valid assumption during the limited simulation time of the work. Furthermore, an uncertainty of $15\%$ on the principal moments of inertia of the target is considered to simulate an imperfect knowledge of the target spacecraft. This situation reflects a scenario in which the shape of an uncooperative target is roughly known, perhaps from a previous inspection phase, yet its inertia properties may not be certain. To ensure the target remains consistently within the cameras’ field of view, a PID controller is employed to control the chaser’s attitude, guaranteeing accurate pointing. The target rotates with an angular velocity of $0.25$ deg/s around each axis. The target is assumed to be in a nearly-circular equatorial orbit, while the relative initial conditions are selected to have a planar and quasi-bounded motion of the chaser with respect to the target. The corresponding relative translational state initial conditions are:

$$\begin{array}{ccc}\hfill {\mathit{\varrho }}_0& =& \left[\begin{array}{ccc}-6.60\times {10}^{-1}& 1.47\times {10}^1& 0.00\end{array}\right]\mathrm{m}\hfill \\ \hfill {\dot{\mathit{\varrho }}}_0& =& \left[\begin{array}{ccc}1.70\times {10}^{-1}& 1.49\times {10}^{-3}& 0.00\end{array}\right]{\mathrm{ms}}^{-1}\hfill \end{array}$$

(11)

Figure 5 reports the evolution of the chaser-target intersatellite distance (right) for the considered trajectory.

The metrics used to assess the estimator performances are position absolute and relative Knowledge Error (KE) and attitude KE. The position KE is defined as in Equations (12) and (13), where $\widehat{\square }$ indicates the estimated values.

$$e_{\mathit{\varrho }}=\parallel \widehat{\mathit{\varrho }}-\mathit{\varrho }\parallel$$

(12)

$$e_{\rho }^{rel}={\displaystyle \frac{e_{\mathit{\varrho }}}{\parallel \mathsf{\rho }\parallel }}$$

(13)

By defining as ${\mathit{q}}_{err}$ and $q_{err}$ the vectorial and scalar part of the quaternion error (i.e., the quaternion representing the discrepancy between the true and estimated attitude), the attitude KE is defined as in Equation (14).

$$e_q=2\mathrm{atan}(\parallel {\mathit{q}}_{err}\parallel ,q_{err})$$

(14)

The mean and associated standard deviation of these figures of merit over a period of time are computed as:

$${\mu }_e={\displaystyle \frac{1}{N}}\sum _{i=1}^N{e}_i$$

(15)

$${\sigma }_e=\sqrt{{\displaystyle \frac{1}{N}}\sum _{i=1}^N{(e_i-{\mu }_e)}^2}$$

(16)

where N represents the number of realizations.

5.3. Filter Initialization

The filter’s tuning parameters are detailed in the present section. Each submatrix of both the state covariance matrix and process noise matrix is assumed to be diagonal and isotropic and they are reported in

Table 2. Initial state covariance settings.

Parameter	Value	Unit
${\mathbf{P}}_0,\mathit{\varrho }$	$1\times {10}^0$	m2
${\mathbf{P}}_0,\dot{\mathit{\varrho }}$	$1\times {10}^{-2}$	m2/s2
${\mathbf{P}}_0,a$	$5\times {10}^{-3}$	-
${\mathbf{P}}_0,\mathit{\omega }$	$1\times {10}^{-1}$	rad2/s2

and

Table 3. Process noise settings.

Parameter	Value	Unit
${\sigma }_{\mathit{\varrho }}$	$1.41\times {10}^{-1}$	m
${\sigma }_{\dot{\mathit{\varrho }}}$	$1.00\times {10}^{-3}$	m/s2
${\sigma }_{\mathit{a}}$	$9.50\times {10}^{-3}$	-
${\sigma }_{\mathit{\omega }}$	$1.00\times {10}^{-3}$	rad/s

, respectively. Please notice that these settings will be kept throughout all the simulations presented in the remainder of the paper. The process noise covariance adopted in this work is meant to compensate for the imperfect dynamical model of the filter and it has been tuned accordingly.

Since the algorithm does not include a pose acquisition routine, such as the ones presented in [46] or [47], the initial pose information is randomly generated by sampling within the $3\sigma$ bounds of the initial covariance. Given this initial pose information, the re-initialization algorithm, described in Algorithm 1, is executed to retrieve the first matching information. The navigation chain is designed to work at 1 Hz.

6. Results

To assess the advantages and drawbacks of the three different sensing modalities, two different test cases are introduced, as reported in

Table 4. Test plan summary.

Test Number	Duration	Objective
1	3000 s	To evaluate the navigation chain using only the VIS images in good illumination conditions.
2	3000 s	To highlight the benefits of the sensor handover data fusion strategy.

. The first test case represents an ideal situation, in which the SAA is forced to be near zero to achieve favourable illumination conditions. It is also employed as a reference for the next simulations, in which the SAA increases throughout the trajectory. It is worth noticing that the thermal infrared images do not change, since they do not rely on external illumination sources. The temperature profile of the spacecraft has been computed apriori, and it is assumed to be constant during the short simulation time.

For each test case, the results of three different sensing strategies are computed:

• VIS-only
• TIR-only
• VIS-TIR fusion with sensor handover

The filter settings shown in the previous section are kept constant for both test cases, and that a total of 250 Montecarlo runs are performed for each sensing strategy.

6.1. Test Case n.1

Representative VIS images acquired at $t=0$ and $t=1000$ s, respectively, are shown in Figure 6.

The averaged results for the relative position (left) and attitude (right) errors are shown in Figure 7. It can be immediately noticed that there is hardly no difference between the combined (VIS and TIR) sensing strategy and the VIS-only sensing mode. This behavior is mainly due to two reasons. The first one is that the trigger to discard VIS images is never activated; the second one is that the filter tends to prioritize the most accurate source of measurement, that in this case is the visible spectrum, thus prioritizing these measurements within the filter update step. The quantitative results are reported in

Table 5. Position and attitude Mean Knowledge Errors (MKE) for different sensing modalities. Test case n.1.

Test Case	Position Errors		Attitude Errors
	Abs. MKE [m]	Rel. MKE [%]	MKE [deg]
VIS	$0.19\pm 0.10$	$1.25\pm 0.68$	$1.10\pm 0.97$
TIR	$0.23\pm 0.15$	$1.50\pm 0.99$	$1.83\pm 1.28$
VIS—TIR Handover	$0.21\pm 0.08$	$1.39\pm 0.49$	$0.94\pm 0.58$

. Once again, it can be noticed that the VIS and the combined sensing modality achieve similar results. It should be acknowledged that the attitude estimation error achieved by the combined sensing modality is the best among the three. Furthermore, the uncertainty of the estimation error tends to decrease when relying on both cameras; even though the number of montecarlo runs is the same. The TIR-based navigation strategy instead shows a higher position and attitude error, together with a higher variance associated to these results. This is due to the fact that a reduced sensor size and a high noise value of the thermal images negatively affects the accuracy of the feature detection and tracking.

To further analyze the presented results, we report in Figure 8 the number of matched feature pairs and the average measurement covariance noise value that the online adaptation of the measurements’ covariance assigns to each camera. The reported value has been averaged considering all the features detected for each step to exclude outliers, and then it has been normalized over the camera size; since the VIS and TIR cameras have a different array size. It can be immediately noticed that the number of matched feature pairs is comparable between VIS and TIR images. This result can be expected due to the good illumination conditions of this test set. It can also be noticed that there is a drop in the number of matched features between time $t=1000$ s and $t=1500$ s, which coincides with an increase of the relative pose estimation error. Since this behaviour is present across both spectra, it can be assumed that it is linked to the relative target-chaser attitude, which does not allow for a higher number of matches. Concerning the mean covariance value assigned to the different spectra, it is immediate to notice that the thermal features are generally more noisy with respect to the visible ones. This result is expected due to the reduced array size and higher noise of thermal infrared cameras. Furthermore, whenever the covariance adaptation produces an increase in the measurement noise covariance, the overall state estimation is affected, as in the period between $t=2000$ s and $t=2500$ s. It can be also acknowledged that whenever the number of tracked features decreases, the covariance adaptation is less stable, since the value of outlier matches may have a greater influence on the overall noise estimation.

6.2. Test Case n.2

The second testing campaign is meant to highlight the flexibility of the developed navigation architecture, by enabling the sensing modality switch. To stress the navigation chain, the simulation lasts 3000 s, during which the illumination conditions change dynamically. In this simulation, the sun aspect angle increases progressively, until the target encounters an eclipse at $t=900$ s approximately, forcing the switch to rely only on TIR images. The evolution of the SAA is portrayed in Figure 9, while two representative VIS images acquired at $t=0$ and $t=1000$ s are shown in Figure 10.

The averaged results over the Montecarlo runs are shown in Figure 11 for the three different sensing modalities, while Figure 12 reports the results for the combined VIS-TIR sensing strategy, in which the utilization of both camera is highlighted in green, and the periods during which the algorithm relies on ther TIR camera only are highlighted in red.

It is evident from the results presented in Figure 11 that visible images alone cannot provide a reliable navigation solution when the SAA increases. The navigation error increases until divergence before 1000 s, due to the fact that the filter can no longer detect and track any features. On the other hand, when the navigation chain relies solely on the thermal imaging measurements the errors present a more erratic behaviour and the pose estimation error tends to increase. This behavior is evident when comparing the results achieved through the sensor handover controlled by the switch, as illustrated in Figure 12, with those derived from using the thermal camera independently, as depicted in Figure 11. Notably, both the position and attitude errors are consistently lower only when the algorithm manages to fuse the visible and thermal information (green area of Figure 12), while they are coincident when the thermal camera is used independently. Analyzing the average errors presented in

Table 6. Position and attitude Mean Knowledge Errors (MKE) for different sensing modalities. Test case n.2.

Test Case	Position Errors		Attitude Errors
	Abs. MKE [m]	Rel. MKE [%]	MKE [deg]
TIR	$0.23\pm 0.15$	$1.50\pm 0.99$	$1.83\pm 1.28$
VIS—TIR Handover	$0.22\pm 0.14$	$1.43\pm 0.96$	$1.54\pm 1.16$

, it becomes evident that, although there is a slight decrease in attitude error, the benefits of employing an adaptive sensing modality solution are enhances during intervals of favorable illumination, while it does not extend to the overall performance for extended periods of dynamical illumination conditions. Figure 11 depicts the estimation error when using the visible spectrum as a standalone solution. As anticipated, the pose estimation abruptly diverges when the image processing pipeline fails to accurately identify and match the target’s features due to the images degradation. This further underscores the critical role of the thermal camera in ensuring the robustness of the navigation chain.

In Figure 11 it can be observed that in both the thermal and sensor-handover scenarios the attitude errors spikes at certain time-steps. This reduction in accuracy occurs when the camera orientation coincides with one of the principal planes of the target, leading to a loss of information regarding its three-dimensional shape. Although this behavior of the navigation chain requires for further investigation on the image processing techniques, it does not significantly impact the pose estimation capability over extended periods.

In the same way as the previous test section, we perform an analysis on the number of matched features and their associated measurement noise covariance. Figure 13 reports the number of matched feature pairs and the average measurement covariance noise value assigned to each camera. First of all, it can be noticed that the number of matched features for the visible spectrum is always lower than the number of thermal-infrared features. In addition, when the visible camera is re-introduced within the navigation chain, its initial covariance value is higher, and it decreases as the feature detection and tracking become reliable again. Despite this difference, the accuracy of VIS-only navigation in favourable illumination conditions is higher with respect to the thermal-infrared case. This is due to the fact that the most important factor is the associated feature noise, which is lower in the case of VIS images. The only time during which VIS features are noisier is in the proximity of eclipses, and the online covariance adaptation assigns a high measurement noise to such features. As for the number of matched thermal features, it can be noticed that the error spikes in the pose estimation error are linked to those periods in which the number of matched feature pairs decreases. This behaviour implies that due to the lower detection and tracking accuracy, thermal-based navigation needs to rely on a higher number of features to obtain accurate results. As expected, the measurement noise covariance of TIR features is higher with respect to VIS ones, since the detection and tracking process is less accurate. It is also necessary to acknowledge that towards the end of the simulation, the measurement noise covariance of TIR features tends to be higher, which is immediately reflected in the attitude error. It is also necessary to point out that the attitude estimation error is more sensitive with respect to the position estimation, since it is tightly linked to the behaviour of the measurements.

7. Conclusions

We present a navigation chain for estimating the relative pose between a chaser and an uncooperative target using combined visible and thermal sensing. The proposed navigation strategy allows for a satisfactory estimation of the relative state and relative angular velocity, and it highlights the two novel aspects of this work: the additional use of a thermal-infrared camera to complement the existing sensor in the relative navigation task and sensor handover between cameras operating in different part of the spectrum. The numerical evaluation of the presented approach confirms that the information added by the thermal camera improves the robustness of navigation architecture in a challenging scenario with dynamical illumination conditions. However, it is important to highlight that the accuracy of the navigation solution employing only thermal infrared images is lower with respect to the one obtained by using visible images in favourable illumination conditions. This outcome is due to two major factors: first of all TIR images have a lower array size and a higher noise level with respect to VIS ones, thus affecting the feature detection and tracking process; and secondly, the employed IP algorithms were initially developed for VIS images only, and thus they have worse performances on TIR images. In addition, an important new research contribution of the paper is in demonstrating the use of a cross-spectral sensor handover, meaning that the navigation filter employs a combined optical and thermal sensing for a part of the orbit, and it autonomously switches to a thermal-only navigation mode when an eclipse occurs. The outcome of this research represents a step forward towards a flexible navigation strategy capable of dealing with any illumination conditions, enabling autonomous operations with uncooperative resident space objects. A step further in this research field requires the development and tailoring of image processing techniques to thermal infrared images to enhance the accuracy of the navigation solution.