Machine Learning-Based Environment-Aware GNSS Integrity Monitoring for Urban Air Mobility

Abstract

The increasing deployment of unmanned aerial vehicles (UAVs) in urban air mobility (UAM) necessitates robust Global Navigation Satellite System (GNSS) integrity monitoring that can adapt to the complexities of urban environments. The traditional integrity monitoring approaches struggle with the unique challenges posed by urban settings, such as frequent signal blockages, multipath reflections, and Non-Line-of-Sight (NLoS) receptions. This study introduces a novel machine learning-based GNSS integrity monitoring framework that incorporates environment recognition to create environment-specific error models. Using a comprehensive Hardware-in-the-Loop (HIL) simulation setup, extensive data were generated for suburban, urban, and urban canyon environments to train and validate the models. The proposed Natural Gradient Boosting Protection Level (NGB-PL) method, leveraging the uncertainty prediction capabilities of the NGB algorithm, demonstrated superior performance in estimating protection levels compared to the classical methods. The results indicated that environment-specific models significantly enhanced both accuracy and system availability, particularly in challenging urban scenarios. The integration of environment recognition into the integrity monitoring framework allows the dynamic adaptation to varying environmental conditions, thus substantially improving the reliability and safety of UAV operations in urban air mobility applications. This research offers a novel protection level (PL) estimation method and a framework tailored to GNSS integrity monitoring for UAM, which enhances the availability with narrower PL bound gaps without yielding higher integrity risks.

1. Introduction

According to the recent GNSS Market Report from EUSPA [1], aviation and UAV applications currently dominate the market in terms of the number of installed GNSS devices, and this situation is projected to remain the same in the near future. Since most UAV applications are envisioned to operate in a mixture of suburban and urban environments [2], where the accuracy, integrity, availability, and continuity of GNSS can vary widely, ensuring the safety of UAV navigation systems for UAM is crucial. It is anticipated that either certain flight zones would need to be avoided considering the GNSS performance [3], or different means of navigation aiding would need to be active [4] in such environments to ensure the desired navigation performance and reduce the risks. The navigation system should also be able to detect and measure the environment-specific risks to avoid endangering safety.

Navigational integrity monitoring is a generic term that refers to the methods and architectures used to observe the performance and trustworthiness of a navigation system. GNSS integrity monitoring has captivated the attention of countless researchers since it was introduced in 1985 with the Global Positioning System (GPS) Integrity Channel (GIC) approach [5]. The importance of this technology cannot be overstated, as it plays a crucial role in ensuring the accuracy and reliability of satellite navigation systems. With the evolution of the GNSS technology, most issues related to GNSS integrity monitoring have already been addressed [6], and the methods such as Receiver Autonomous Integrity Monitoring (RAIM) and its extended versions [7] have been improved over the decades.

The current GNSS integrity monitoring approaches typically rely on static error models that do not account for the varying environmental conditions encountered in urban areas. This oversight leads to significant performance degradation, increased navigation errors, and higher integrity risks in GNSS-challenging environments. Moreover, the inability of the traditional methods to dynamically adapt to the urban landscape limits their effectiveness in providing the necessary reliability and safety for UAM operations.

Urban integrity monitoring has been researched in the literature, and the challenges, such as signal blockages, multipath reflection, and NLoS receptions, have been addressed [8]. While these works aim to describe the generic integrity criteria for urban environments in order to ensure reliable GNSS performance, environment types or urbanization levels are not taken into account. Considering GNSS performance differentiates based on the environment, some studies examine how GNSS or environment-based parameters such as Dilution of Precision (DOP) coefficients, number of visible satellites, cut-off elevation, and Signal-to-Noise Ratio (SNR) change in different environments [9,10,11]. The results show that while the impact of LoS obscuration on the GNSS accuracy and integrity can be specified with DOP, the number of visible satellites, and cut-off elevation, SNR is mostly useful for detecting multipath effects.

Assuming that the distribution of these parameters and their impact on the error models vary with respect to the environmental conditions, the idea of environment recognition has emerged. An environment recognition algorithm is suggested in [12] to distinguish the five most common urban environment types based on GNSS blockage and signal strength. Another algorithm for determining LoS blockage characteristics around a railway is presented in [13] with clustering GNSS elevation and azimuth. In [14], GNSS SNR data are classified for different environment types based on their elevation angles. Some other studies reviewed in [15] also benefited from multiple onboard sensors for detecting environment types. Nevertheless, the scope of these studies is limited to environment detection or recognition, and their impact on GNSS performance has not been investigated.

In this study, we aimed to develop a novel ML-based GNSS integrity monitoring framework utilizing environment-specific error models for UAM applications. With this aim, initially, environment-specific data were generated via HIL simulation, as described in Section 3. Then, the most suitable environmental and GNSS-based features were specified and extracted from the generated data to train the environment-specific error models. After the error models were trained with the Natural Gradient Boost (NGB) algorithm [16], which is capable of estimating uncertainties of predictions, PLs were estimated with test and validation data to perform an integrity monitoring analysis. Lastly, the estimated NGB-based PLs (NGB-PL) were compared with the PL estimations of the existing approaches for validation. The novel contributions of this study can be listed:

• A novel integrity monitoring framework implementing environment recognition functionalities into the conventional integrity monitoring architecture
• A novel PL estimation method, NGB-PL, is proposed considering the uncertainty prediction capabilities of the NGB algorithm.
• A comprehensive feature importance analysis is conducted to reveal the impact of GNSS and environment-based observables on the error estimation in urban environments.
• Performance assessment and comparison between NGB-PL and other conventional PL estimation methods are presented in various urban scenarios.

The rest of the paper is structured as follows. Section 2 reviews the related works, including integrity monitoring, environment recognition, PL estimation methods, and ML utilization for PL estimations. In Section 3, the proposed integrity monitoring framework is introduced, and the overall research methodology is explained. Section 4 presents the training data and feature analysis results, followed by a performance assessment and comparison of the proposed and existing integrity monitoring approaches. Finally, Section 5 discusses the findings of this study, and a brief conclusion and future works are given in Section 6.

2. Related Works

2.1. GNSS Integrity Monitoring in Urban Environments

GNSS integrity monitoring can be divided into two main categories with respect to its implementation: system-level and user-level [17]. System-level integrity monitoring is performed by Satellite-based Augmentation Systems (SBASs) and Ground-based Augmentation Systems (GBASs) and relies on external measurements, which are provided by augmentation satellites or ground reference stations [18]. On the other hand, user-level integrity is performed internally by the user receiver, where no differential correction or external data are required. Since the user-level integrity monitoring concept was originally introduced for aviation, it is considered under the umbrella of Airborne-based Augmentation Systems (ABASs). ABAS integrity monitoring is broken down into Airborne Autonomous Integrity Monitoring (AAIM), where GNSS-INS (Inertial Navigation System) integration is implemented, and Receiver Autonomous Integrity Monitoring (RAIM), where a stand-alone GNSS is in use [19]. Besides these autonomous methods, Satellite Autonomous Integrity Monitoring (SAIM) has been suggested in [20], where satellites perform integrity monitoring internally to detect and exclude space segment errors. In this study, we focused on offline stand-alone GNSS integrity monitoring at the user level. Therefore, our interest lies specifically in the RAIM variants rather than other autonomous or system-level integrity architectures.

RAIM was initially introduced by Lee in [21] as a foundational technique of user-level autonomous integrity monitoring implemented in GNSS receivers and is still widely used due to its simplicity and effectiveness in providing essential integrity monitoring. RAIM variants have been originally designed for general aviation users, mainly operating at high altitudes and in open-sky environments with negligible amounts of multipath reflections and line-of-sight (LoS) blockages. Specifically in aviation and maritime applications, error models and integrity requirements are well structured, and the existing methods are able to meet the required navigation performance (RNP) [22,23] where the performance criteria are not stringent. Additionally, real-time Integrity Support Messages (ISMs) from various monitoring stations are available thanks to the clear LoS connection in open-sky environments, providing up-to-date information on the integrity risk for different fault modes.

Nevertheless, RAIM approaches cannot address the unique challenges of urban settings, and they cannot guarantee the intended reliability and availability of GNSS-based navigation in urban environments for the following reasons:

• Multipath reflections, non-LoS (NLoS) receptions or complete signal blockages are very common.
• Error models and fault modes structured with respect to the open-sky conditions significantly differ in urban environments.
• Multiple simultaneous faults are prevalent.
• ISM generated by integrity monitoring stations are unavailable as they are also LoS dependent.
• Stricter accuracy and integrity requirements are expected as urban environments are much more constricted and challenging for satellite navigation.

RAIM variants have two commonly used approaches to perform integrity monitoring [24]: the Measurement Rejection Approach (MRA) and the Error Characterization Approach (ECA). The MRA approach is designed to identify and exclude any faulty or degraded measurements from the overall position estimation. This helps to mitigate the negative impact of such measurements. On the other hand, the ECA approach characterizes measurement errors and calculates a protection level (PL) to secure against them. It does not require identifying and removing degraded measurements, even if they contain large errors. Both approaches have their unique strengths and can lead to a similar level of integrity [25].

We have concentrated on enhancing ECA rather than MRA. This is because MRA is generally ineffective in GNSS-challenging environments due to the multiple simultaneous faults and limited signal availability. Even if seamless fault detection and exclusion (FDE) with the MRA were possible in urban environments, excluding all the faulty signals would lead to an insufficient number of signals for position estimations at the receiver level.

Although the FDE functionality is also available with most ECA-based integrity monitoring methods, this is mostly limited to isolating single or several significant faults. This means that PL estimations should be able to deal with the existence of faulty measurements, and their size should be proportional to the effect of the faulty signal. Therefore, we propose a snapshot PL estimator to enhance the ECA functionality for integrity monitoring that is resistant to the dynamic environmental conditions of UAM applications. Before introducing the proposed method, the environmental awareness and PL estimation approaches need to be reviewed.

2.2. GNSS-Based Environmental Awareness

As mentioned earlier, GNSS integrity monitoring systems can benefit from GNSS observables while predicting faulty signals and position errors. Additionally, some environmental parameters can enhance this prediction performance if applicable. However, the GNSS error models are not the same in different sorts of environments. Thus, environmental awareness is necessary to produce better and more accurate error models, especially in urban environments.

Different environment types can be defined based on the density of urban areas. The most common environment types used in the literature are open sky, suburban, urban, urban canyon, and forest, considering the intensity of LoS blockage and multipath reflections [12]. As we are interested in predicting the GNSS performance in urban areas, three urbanization levels were selected and defined based on their cut-off elevation and multipath effects:

• Suburban: A settlement area consisting mostly of 2- or 3-storey houses, with a cut-off elevation angle of 10 degrees. The multipath effect is too low.
• Urban: A settlement area covered with apartments, and the cut-off elevation angle is 10 to 30 degrees. Multipath reflections have more impact than suburban areas.
• Urban canyon: An area surrounded by many tall buildings sheathed with highly reflective material creates a dense multipath impact. The cut-off elevation angle varies between 30 and 60 degrees.

While generating environment-specific data in this study, the environment types were considered as defined above, and error models were trained and tested with this data set.

Based on the definitions of environment types, the key separators between these environments are satellite visibility and the intensity of multipath impact. While DOP, the number of visible satellites, and elevation angle provide information about satellite visibility and geometry, the variation in SNR is beneficial for detecting multipath reflections. These parameters have been used in the literature in different ways.

The first approach is the direct usage of these parameters as extracted by the receiver [13]. In this case, some candidate input features for environment classification or clustering algorithms are as follows:

• DOP;
• Number of visible satellites;
• Elevation angles of satellites;
• Azimuth angles of satellites;
• SNR of satellites.

As all of these features are directly available at the user level without requiring further calculation or comparison than a receiver normally does, this approach is applicable for in-flight environment recognition.

The second approach is to use the relative values of features as input data. In this approach, each feature’s ideal or expected values under the open sky and multipath-free conditions should be known in advance to calculate the differences or rates with respect to these ideal values. The suggested relative environment recognition features are as follows [12]:

• DOP expansion ratio;
• Blockage coefficient;
• Mean and standard deviation of signal strength attenuation;
• Signal strength fluctuation coefficient.

Environment recognition accuracy would be better when the relative features are implemented as input to the classification algorithm. However, this approach is not applicable in a real-time scenario unless a calibration flight is conducted or in-flight conditions are simulated in advance.

Some additional features can also be extracted from additional onboard sensors, such as cameras or Lidar, to enhance the environment recognition accuracy, which causes additional complexity in the system design [15].

Nevertheless, in this study, we were not focused on contributing to the field of environment recognition. Instead, we were integrating an environment recognition function into the integrity monitoring framework to create environment-specific error models and to estimate PLs based on these models. In order to prevent errors caused by the misidentification of environments, the training data were generated separately for each type of environment. The process of generating training data is explained in detail in Section 3.

2.3. Existing PL Estimation Approaches

Basically, the PL should satisfy the condition below, where $\mathsf{\alpha }$ is the target probability of misleading information:

$$\mathrm{P}(\mathrm{P}\mathrm{E}>\mathrm{P}\mathrm{L})<\mathsf{\alpha }$$

(1)

When we consider the fault-free scenario or the stage after the FDE function is applied, the classical PL estimation approach assumes that the error model of the remaining measurement errors converges to a zero-mean Gaussian model. Thus, the typical equation of classical approach, which is adopted by RAIM variants, SBASs, and GBASs, can be defined as,

$$\mathrm{x}\mathrm{P}\mathrm{L}=\mathrm{K}·{\mathsf{\sigma }}_{\mathrm{x}}·\mathrm{x}\mathrm{D}\mathrm{O}\mathrm{P}$$

(2)

where x refers to the horizontal or vertical direction, σ is the error variance, and K is the Gaussian distribution confidence coefficient for overbounding the target integrity risk (IR), $1-\mathsf{\alpha }$. Equation (2) can be explicitly rearranged to horizontal and vertical axes as suggested in [26],

$${\mathrm{H}\mathrm{P}\mathrm{L}}_{\mathrm{S}\mathrm{B}\mathrm{A}\mathrm{S}}={\mathrm{K}}_{\mathrm{H}}·{\mathrm{d}}_{\mathrm{m}\mathrm{a}\mathrm{j}\mathrm{o}\mathrm{r}},\hspace{1em}\hspace{1em}\hspace{1em}{\mathrm{V}\mathrm{P}\mathrm{L}}_{\mathrm{S}\mathrm{B}\mathrm{A}\mathrm{S}}={\mathrm{K}}_{\mathrm{V}}·{\mathrm{d}}_{\mathrm{u}}$$

(3)

where,

$${\mathrm{d}}_{\mathrm{m}\mathrm{a}\mathrm{j}\mathrm{o}\mathrm{r}}=\sqrt{{\displaystyle \frac{{\mathrm{d}}_{\mathrm{E}}^2+{\mathrm{d}}_{\mathrm{N}}^2}{2}}+\sqrt{{\left({\displaystyle \frac{{\mathrm{d}}_{\mathrm{E}}^2-{\mathrm{d}}_{\mathrm{N}}^2}{2}}\right)}^2+{\mathrm{d}}_{\mathrm{E}\mathrm{N}}^2}}$$

(4)

$$\mathrm{D}={{(\mathrm{G}}^{\mathrm{T}}\mathrm{W}\mathrm{G})}^{-1}=\left[\begin{array}{cccc}{\mathrm{d}}_{\mathrm{E}}^2& {\mathrm{d}}_{\mathrm{E}\mathrm{N}}& {\mathrm{d}}_{\mathrm{E}\mathrm{U}}& {\mathrm{d}}_{\mathrm{E}\mathrm{T}}\\ {\mathrm{d}}_{\mathrm{E}\mathrm{N}}& {\mathrm{d}}_{\mathrm{N}}^2& {\mathrm{d}}_{\mathrm{N}\mathrm{U}}& {\mathrm{d}}_{\mathrm{N}\mathrm{T}}\\ {\mathrm{d}}_{\mathrm{E}\mathrm{U}}& {\mathrm{d}}_{\mathrm{N}\mathrm{U}}& {\mathrm{d}}_{\mathrm{U}}^2& {\mathrm{d}}_{\mathrm{U}\mathrm{T}}\\ {\mathrm{d}}_{\mathrm{E}\mathrm{T}}& {\mathrm{d}}_{\mathrm{N}\mathrm{T}}& {\mathrm{d}}_{\mathrm{U}\mathrm{T}}& {\mathrm{d}}_{\mathrm{T}}^2\end{array}\right]$$

(5)

The weighted directional covariance matrix, D, can be derived from the weighting matrix, $\mathrm{W}$, and the geometry matrix, G, which are defined below:

$$\mathrm{G}=\left[\begin{array}{cccc}-\mathrm{c}\mathrm{o}\mathrm{s}{\mathsf{\theta }}_1\mathrm{s}\mathrm{i}\mathrm{n}{\mathsf{\phi }}_1& -\mathrm{c}\mathrm{o}\mathrm{s}{\mathsf{\theta }}_1\mathrm{c}\mathrm{o}\mathrm{s}{\mathsf{\phi }}_1& -\mathrm{s}\mathrm{i}\mathrm{n}{\mathsf{\theta }}_1& 1\\ -\mathrm{c}\mathrm{o}\mathrm{s}{\mathsf{\theta }}_2\mathrm{s}\mathrm{i}\mathrm{n}{\mathsf{\phi }}_2& -\mathrm{c}\mathrm{o}\mathrm{s}{\mathsf{\theta }}_2\mathrm{c}\mathrm{o}\mathrm{s}{\mathsf{\phi }}_2& -\mathrm{s}\mathrm{i}\mathrm{n}{\mathsf{\theta }}_2& 1\\ -\mathrm{c}\mathrm{o}\mathrm{s}{\mathsf{\theta }}_3\mathrm{s}\mathrm{i}\mathrm{n}{\mathsf{\phi }}_3& -\mathrm{c}\mathrm{o}\mathrm{s}{\mathsf{\theta }}_3\mathrm{c}\mathrm{o}\mathrm{s}{\mathsf{\phi }}_3& -\mathrm{s}\mathrm{i}\mathrm{n}{\mathsf{\theta }}_3& 1\\ ⋮& ⋮& ⋮& ⋮\end{array}\right]$$

(6)

where ${\theta }_i \mathrm{a}\mathrm{n}\mathrm{d} {\phi }_i$ are the elevation and azimuth of the $i^{th}$ satellite.

$$\mathrm{W}=\left[\begin{array}{cccc}{\mathrm{w}}_1& 0& \cdots & 0\\ 0& {\mathrm{w}}_2& \cdots & 0\\ ⋮& ⋮& \ddots & ⋮\\ 0& 0& \cdots & {\mathrm{w}}_{\mathrm{n}}\end{array}\right],\hspace{1em}\hspace{1em}\hspace{1em}{\mathrm{w}}_{\mathrm{i}}={\displaystyle \raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{${\mathsf{\sigma }}_{\mathrm{i}}$}\right.} \mathrm{f}\mathrm{o}\mathrm{r} {\mathrm{i}}^{\mathrm{t}\mathrm{h}} \mathrm{s}\mathrm{a}\mathrm{t}\mathrm{e}\mathrm{l}\mathrm{l}\mathrm{i}\mathrm{t}\mathrm{e}$$

(7)

The primary difficulty with the classical approach lies in estimating overbounding sigma values to calculate PLs when the actual error model is not a zero-mean Gaussian. To ensure that the error model overbounds the actual error distribution, basically, two approaches can be implemented [27]. The first one is broadcasting inflated sigma values for each satellite based on the presumed error models from a reference station. However, since we are seeking a solution applicable to UAM, a method requiring this sort of online updating does not seem to be feasible due to the limited availability of ISM data. The second one involves using mathematical models of error sources to calculate inflated sigma values based on GNSS observables within the user receiver. Overbounding pseudorange error variances for each non-faulty satellite, ${\mathsf{\sigma }}_{\mathrm{i}}$, can be modelled for L1 single-frequency users [26],

$${\mathsf{\sigma }}_{\mathrm{i}}^2={\mathsf{\sigma }}_{\mathrm{i},\mathrm{U}\mathrm{R}\mathrm{A}}^2+{\mathsf{\sigma }}_{\mathrm{i},\mathrm{U}\mathrm{I}\mathrm{R}\mathrm{E}}^2+{\mathsf{\sigma }}_{\mathrm{i},\mathrm{t}\mathrm{r}\mathrm{o}\mathrm{p}\mathrm{o}}^2+{\mathsf{\sigma }}_{\mathrm{i},\mathrm{a}\mathrm{i}\mathrm{r}}^2$$

(8)

• ${\mathsf{\sigma }}_{\mathrm{i},\mathrm{U}\mathrm{R}\mathrm{A}}^2$ is the user range accuracy and is accepted as 0.75 m and 0.67 m for GPS and Galileo satellites, respectively.
• ${\mathsf{\sigma }}_{\mathrm{i},\mathrm{U}\mathrm{I}\mathrm{R}\mathrm{E}}^2$ is ionospheric delay estimation error variance, which can be calculated as a function of ionospheric delay (${∆\mathsf{\tau }}_{\mathrm{i}\mathrm{o}\mathrm{n}\mathrm{o}}$) [28] and obliquity factor (${\mathrm{F}}_{\mathrm{p}\mathrm{p}}$) [26] as below:

  $${\mathsf{\sigma }}_{\mathrm{U}\mathrm{I}\mathrm{R}\mathrm{E}}={\mathrm{F}}_{\mathrm{p}\mathrm{p}}\times {∆\mathsf{\tau }}_{\mathrm{i}\mathrm{o}\mathrm{n}\mathrm{o}}$$

  (9)
• ${\mathsf{\sigma }}_{\mathrm{i},\mathrm{t}\mathrm{r}\mathrm{o}\mathrm{p}\mathrm{o}}^2$ is the tropospheric error variance and can be calculated based on a tropospheric delay model mentioned in [29] with Black and Eisner’s mapping function [30], which is valid for the satellites with more than a 4° elevation angle:

  $${\mathsf{\sigma }}_{\mathrm{i},\mathrm{t}\mathrm{r}\mathrm{o}\mathrm{p}\mathrm{o}}^2=0.12\times \left({\displaystyle \frac{1.001}{0.002001+{\sin }^2{\theta }_i}}\right)$$

  (10)
• ${\mathsf{\sigma }}_{\mathrm{i},\mathrm{a}\mathrm{i}\mathrm{r}}^2$ is the variance of airborne (user) receiver error, which is composed of noise and multipath terms to represent receiver-based errors [31].

$${\mathsf{\sigma }}_{\mathrm{i},\mathrm{a}\mathrm{i}\mathrm{r}}^2={\mathsf{\sigma }}_{\mathrm{i},\mathrm{n}\mathrm{o}\mathrm{i}\mathrm{s}\mathrm{e}}^2+{\mathsf{\sigma }}_{\mathrm{i},\mathrm{m}\mathrm{p}}^2$$

(11)

for Galileo satellites, a lookup table is provided by [31] for ${\sigma }_{i,air}^2\equiv {\sigma }_{i,user}^2$.

for GPS satellites:

$${\mathsf{\sigma }}_{\mathrm{i},\mathrm{n}\mathrm{o}\mathrm{i}\mathrm{s}\mathrm{e}}^2=0.15+0.43{\mathrm{e}}^{\left({\mathsf{\theta }}_{\mathrm{i}}/6.9°\right)}$$

(12)

$${\mathsf{\sigma }}_{\mathrm{i},\mathrm{m}\mathrm{p}}^2=0.13+0.53{\mathrm{e}}^{\left({\mathsf{\theta }}_{\mathrm{i}}/10°\right)}$$

(13)

The accuracy of classical PL estimations strictly relies on predefined error models, and external ISM data are mostly required for real-time applications. In order to enhance the offline integrity monitoring capabilities, GMV introduced a patented isotropy-based PL (IBPL) estimation method [32].

IBPL emerged from the centred Gaussianity assumption of the classical PL approach [33]. Considering this assumption, the norm of LSE residuals, r, should satisfy the equation below with respect to the number of measurements, N, used for the solution.

$$\left|\left|r\right|\right|=\sigma ·\sqrt{N-4}$$

(14)

If (14) is written in (2), the overbounding error variances can be replaced with LSE residuals.

$${xPL}_{IBPL}=K·{\displaystyle \frac{\left|\left|r\right|\right|}{\sqrt{N-4}}}·xDOP$$

(15)

After introducing a new isotropy-based confidence coefficient, k, and assuming,

$$k·\sqrt{N-4}\underset{N\to \infty }{\to }K$$

(16)

IBPL can be defined as:

$$xPL=k·\left|\left|r\right|\right|·xDOP$$

(17)

As seen from (16), k varies with changing N values and can be computed numerically by solving the following equation for any intended $\mathsf{\alpha }$ value,

$$\mathsf{\alpha }={\displaystyle \frac{\mathrm{N}-2}{2}}·{\left(1+{\mathrm{k}}^2\right)}^{{\displaystyle \frac{4-\mathrm{N}}{2}}}-{\displaystyle \frac{\mathrm{N}-4}{2}}·{\left(1+{\mathrm{k}}^2\right)}^{{\displaystyle \frac{2-\mathrm{N}}{2}}}$$

(18)

IBPL-based integrity monitoring cannot offer any FDE function, but its capability to perform integrity monitoring without any error model assumption is unique. However, this approach is sensitive to decreased satellite visibility and large measurement errors [24]. Hence, the IBPL efficiency may be affected in urban areas where the satellite visibility is low and the measurement errors are large.

GMV has also introduced another patented method called Kalman Integrated Integrity Monitoring (KIPL) [34], which offers a sequential PL estimation approach, especially for GNSS-INS integrated navigation systems. However, this method was not investigated in the scope of this study since we focused on snapshot algorithms for stand-alone GNSS.

2.4. Machine Learning Utilization for PL Estimations

ML utilization in the field of GNSSs has expanded significantly across various applications such as signal detection and classification, error prediction, precise positioning, etc. [35]. However, the accuracy of PL estimations relies heavily on the precise estimation of error variances. The traditional regression models often fail to provide reliable estimates of these overbounding error variances and do not adequately support the integrity monitoring in urban environments. Thus, there is a need for probabilistic approaches that can deliver a complete probability distribution of the potential outcomes. Unlike the conventional regression models, which provide a single-point prediction, these probabilistic regression models offer valuable insights into the uncertainty surrounding predictions.

Various methods have been developed for probabilistic regression, each with unique strengths and limitations. The classical regression models can be extended to probabilistic settings, but these adaptations often assume that the variance remains constant across all instances, a limitation known as homoscedasticity [16]. Generalized Additive Models for Location, Scale, and Shape (GAMLSS) [36] address this by allowing the variance to vary with the input features, accommodating heteroscedasticity, but they are limited to pre-specified forms of the distribution, which may not always suit complex data patterns. Bayesian methods, including Bayesian Neural Networks [37] and Bayesian Additive Regression Trees (BARTs) [38], provide a principled approach to uncertainty by integrating predictions over the posterior distribution. However, the flexibility of the Bayesian methods often comes at the cost of computational complexity, as inference requires approximation techniques which are computationally demanding and require expertise to implement effectively. Bayesian Deep Learning with probabilistic backpropagation [39] tries to combine deep learning with Bayesian inference to estimate uncertainty, yet this method also has limitations, especially when data are structured or tabular and limited in volume.

Gradient Boosting Machines (GBMs) [40] present another promising approach, offering a powerful, flexible, and relatively easy-to-use solution for many regression problems. However, standard GBMs produce scalar outputs in regression tasks, typically yielding only point estimates. Some adaptations interpret these outputs as the mean of a Gaussian distribution but assume constant variance, reducing the utility of this approach for applications requiring a fuller understanding of the predictive uncertainty. Moreover, extending GBMs to produce multiple parameters (for example, both mean and variance) presents optimization challenges, as each parameter has different sensitivities and dynamics. Without modification, traditional GBMs may struggle to capture the desired range of probabilistic outcomes, especially for data with highly variable uncertainty.

Natural Gradient Boosting (NGB) [16] has emerged as an advancement over these existing methods, particularly by addressing the weaknesses in the traditional GBMs when applied to probabilistic tasks. NGB expands the gradient boosting framework to probabilistic regression by estimating multiple distribution parameters simultaneously, allowing for comprehensive probabilistic predictions that consider both the central tendency and uncertainty. A key innovation in NGB is the use of the natural gradient instead of the ordinary gradient [41]. The natural gradient corrects the issues in standard gradient optimization by taking into account the geometric structure of the parameter space, leading to more balanced and stable updates across the parameters. This correction is particularly advantageous in probabilistic settings, as it helps balance the updates for different parameters, such as the mean and variance, thereby improving the model robustness and convergence stability.

The modular design of NGB further enhances its utility, allowing users to select the distributional family, scoring rule, and base learners most suitable for their data. This flexibility allows NGB to adapt to a wide range of applications without requiring extensive customization.

3. Methodology

3.1. The Proposed Integrity Monitoring Framework

Probabilistic estimation is essential for integrity monitoring where predicting outcomes with associated uncertainties is crucial. The existing methods for probabilistic regression offer valuable insights but often require complex modelling or make restrictive assumptions about the data variability. NGB offers a practical and flexible solution that leverages natural gradients to address these limitations, providing a comprehensive and accessible approach to probabilistic regression. Moreover, the test results provided in [16] show that NGB is one of the best-performing algorithms in both single-point and probabilistic regression. Therefore, in this study, the NGB-PL approach is proposed by utilizing the NGB algorithm for PL estimations. According to the error prediction results in [42], a decision tree algorithm was chosen as the base learner for NGB.

No matter how well the NGB algorithm performs, the success of the NGB-PL approach depends on correctly training the error models with sufficient data. Suppose that the algorithm is trained with generic data regardless of the environment. In that case, the error model is expected to provide large uncertainties for error predictions, resulting in very conservative PL estimations. For this reason, an environment recognition unit should be integrated into the framework to benefit from environment-specific error models. The proposed integrity monitoring framework is illustrated in Figure 1.

3.2. HIL Simulation Setup

In the given framework, error and uncertainty prediction units construct the NGB estimator together. PLs are later calculated using error and uncertainty predictions. The environment recognition unit initiates the switch between the predefined environments to enable the relevant error model in the NGB estimator. Since our main focus was to analyze the accuracy of error and uncertainty predictions, the environment recognition unit was assumed to provide fault-free classification results, and its development was beyond the scope of this study.

The FDE unit is designed to detect and exclude faulty signals if possible. In practice, most low-cost receivers can perform basic RAIM FDE to eliminate one faulty signal per epoch. So, we assumed that the FDE capability of the u-blox receiver represents the typical functionality of low-cost GNSS receivers. Therefore, no further development of the FDE unit is proposed, as the receiver already applies FDE.

For the characterization of the GNSS error sources to support integrity monitoring, we need to build deterministic threat models. However, the exact error bounds are unknown due to the uncertainties in the transmission environment. In order to develop realistic GNSS error models and understand how GNSS observables affect the navigation performance, a large amount of data is required. GNSS almanac data provide observations and orbital information about all satellites and can be found on publicly available databases such as the Crustal Dynamics Data Information System (CDDIS) [43], where national or international programs acquire the data. This kind of data is helpful for the reproducibility of GNSS measurements with software-defined receivers (SDRs), particularly when open sky scenarios are considered. Nevertheless, it is important to note that analyses conducted with almanac data may not be valid in real-life scenarios, especially for urban environments, since the dynamic uncertainties of the environment, transmission line, or receiver are ignored.

Collecting reliable data for statistical analysis can also be accomplished by building an experimental setup. Although using experimental data may sound more appealing than relying on almanac data, setting the environmental features to meet our test requirements can be challenging. During flight experiments, GNSS observables provide limited information about the environment type and signal quality, which is crucial for our environment-aware integrity monitoring approach. Even if it is possible to overcome this challenge, another issue arises when dealing with the atmospheric and ionospheric error sources since we have no control over them. Lastly, generating hours of continuous experimental data is neither practical nor useful when the required amount of data is taken into consideration for statistical analysis in this research. Thus, in this study, a HIL simulation setup was chosen to ensure the complete control over all the aspects of GNSS navigation while maintaining a high level of realism.

The HIL simulation structure is illustrated in Figure 2, which comprises Spirent’s GSS7000 GNSS Constellation Simulator [44] with its simulation software SimGen [45] and a commercial-level low-cost GNSS receiver, u-blox ZED-F9P [46]. Although alternative GNSS signal generators are available on the market, especially for commercial use, Spirent’s setup was chosen for its accessibility in our laboratory and its ability to simulate urban environment scenarios.

The SimGen GNSS simulation software offers a wide range of options for defining the scenarios and parameters related to all the GNSS segments. When using SimGen, GSS7000 can generate navigation messages and create a navigation environment with all the necessary details, e.g., the position of the satellites and receiver, potential faults, tropospheric or ionospheric effects, obstructions, and signal reflections. By creating a proper scenario in the GSS7000 simulator, it is possible to observe the independent impact of each error source on the position measurement realistically.

The RF signals generated by the GSS7000 simulator are capable of replicating the real GNSS signals under the defined conditions in the SimGen software. These signals can be used as the antenna input for any GNSS receiver. While the GSS7000 simulator generates GNSS messages, SimGen creates log files at the same time, including the details of the messages and simulation features. NMEA and RINEX data files are also available from the simulator. All of this information was considered as the ground truth throughout this research. The rest of the data were sourced from the employed receiver.

Basically, our setup had three physical data monitoring points: the RF output of the GSS7000 simulator, raw measurements at the input of the GNSS receiver, and the receiver’s output with PVT estimations. In order to extract the useful information from the different types of data records, we first had to convert them into a MATLAB-compatible format. To achieve this, we converted the simulator’s log files and the receiver’s RINEX files into the .csv format, which made them easily readable by MATLAB. However, extracting the data from NMEA files required more effort, as each line was allocated to a specific talker and message type identifier. Therefore, we fragmented the NMEA folders into subfolders and saved them as individual .csv files, each containing particular types of messages only.

Once the data extraction was completed, the data preprocessing stage had two main challenges: ensuring proper synchronization and conducting data health checks. The timestamps on the receiver’s recordings were rounded due to the restricted resolution capabilities of the NMEA and RINEX formats. However, the exact timestamps can be calculated by taking into account the receiver’s clock bias, which is available at the receiver’s interface. After the clock corrections have been made on the timestamps, the mismatch between the receiver’s recordings and the ground truth can be resolved.

To conduct data health checks, we developed a systematic approach to check the quality and completeness of the available data. Our first step was to verify the availability of the position estimation by the receiver, commonly called a 3D fix. If a 3D fix did not occur for a particular timestamp, we excluded all the information associated with that timestamp from our data set. Additionally, data samples with any missing information were also excluded from the data set. These helped us to ensure that we were only analyzing reliable data. By doing so, we could prevent any errors or inaccuracies that might occur due to the use of incomplete or misleading information.

4. Results

4.1. Simulation

Having discussed the methodology, we will initially explain the details of the simulation strategy, including the environment models and scenarios built in order to generate data to train the environment-specific error models properly. Training data were generated separately for the three urban environment types (suburban, urban, and urban canyon), which were simulated using SimGen software. These environment types are defined based on the assumptions in Section 2.2. Their signal blockage and reflection characteristics are illustrated in Figure 3, Figure 4 and Figure 5 with respect to the elevation and azimuth angles of satellites.

During the simulation period, satellites’ skyplot changed gradually due to the movement of the satellites and the vehicle. Thus, the impact of varying environments on the satellite visibility and geometry is discussed in Section 4.2 as a part of the training data analysis.

For each environment model, 24 h long training data, which began at midnight on 23 January 2022, were generated in the GSS7000 GNSS simulator. During the simulations, the UAV maintained a constant altitude of 50 m and a constant speed of 10 m/s, and the trajectory is illustrated in Figure 6. The starting location is (0,0,0) in the LLA (Longitude–Latitude–Altitude) coordinate system.

The data consisted of single-frequency (L1) GPS and Galileo signals, which were processed by a low-cost GNSS receiver, u-blox ZED-F9P. This receiver is equipped with onboard RAIM FDE capabilities to detect and exclude one faulty measurement per epoch. Position errors were calculated from the difference between the receiver’s position estimation and the ground truth of the simulator. The receiver also records GNSS-related information in NMEA and RINEX data formats, and input features were extracted from these records.

Twenty per cent of the generated data for each environmental scenario were randomly allocated for testing. The rest of the data were utilized for training. Additional 3 h long continuous validation data were also generated at midnight the next day. After training, testing, and validation data allocations were performed, the following input features were considered for training the NGB algorithm to generate environment-specific error models:

• Number of visible satellites (vS);
• Minimum, mean, and standard deviation of the satellite elevations (${\mathrm{m}\mathrm{i}\mathrm{n}}_{\mathrm{e}\mathrm{l}\mathrm{e}}$/${\mathsf{\mu }}_{\mathrm{e}\mathrm{l}\mathrm{e}}$/${\mathsf{\sigma }}_{\mathrm{e}\mathrm{l}\mathrm{e}}$);
• Mean and standard deviation of the satellite azimuths (${\mathsf{\mu }}_{\mathrm{a}\mathrm{z}\mathrm{i}}$/${\mathsf{\sigma }}_{\mathrm{a}\mathrm{z}\mathrm{i}}$);
• HDOP/VDOP;
• Mean and standard deviation of satellite Signal-to-noise ratios (${\mathsf{\mu }}_{\mathrm{s}\mathrm{n}\mathrm{r}}$/${\mathsf{\sigma }}_{\mathrm{s}\mathrm{n}\mathrm{r}}$);
• Directional covariances from Equation (4) (${\mathrm{d}}_{\mathrm{m}\mathrm{a}\mathrm{j}\mathrm{o}\mathrm{r}}$/${\mathrm{d}}_{\mathrm{u}}$);
• Norm of Least-squared estimation (LSE) residuals ($\left|\right|\mathrm{r}\left|\right|$).

The response of our trained model was position error estimations and error variances which are necessary for PL estimations. In order to enhance the training performance, the input features were normalized with respect to their mean (${\mathrm{\mu }}_{\mathrm{x}}$) and standard deviation (${\mathsf{\sigma }}_{\mathrm{x}}$), as below:

$${\mathrm{X}}_{\mathrm{n}\mathrm{o}\mathrm{r}\mathrm{m}}=\left(\mathrm{X}-{\mathrm{\mu }}_{\mathrm{x}}\right)/{\mathsf{\sigma }}_{\mathrm{x}}$$

(19)

4.2. Training Data Analysis

In this section, we will analyze the training data generated via the HIL simulation with respect to the environment types. This analysis gives an insight into how the environment types affect the position accuracy, satellite visibility, and satellite geometry. Multipath-free scenarios for urban and urban canyon environments are also analyzed to see the impact of the multipath reflections and NLOS receptions on the accuracy of the position estimations.

Table 1. Training data statistics.

Environment Type	Horizontal RMSE (m)			Vertical RMSE (m)			HDOP	VDOP	vS	Availability (%)
	µ	σ	95th Percentile	µ	σ	95th Percentile	µ	µ	µ
Open sky	0.26	0.19	0.66	0.35	0.26	0.80	0.59	1.11	19.27	100
Suburban	0.27	0.24	0.67	0.41	0.44	0.90	0.60	1.11	19.25	100
Urban	2.40	1.59	5.59	8.34	5.14	17.50	0.71	1.30	16.35	100
Urban Multipath-free	0.25	0.17	0.60	0.34	0.23	0.78	0.73	1.32	15.63	100
Urban canyon	11.10	8.39	26.83	35.66	20.34	75.90	0.90	1.47	12.65	100
Urban canyon Multipath-free	1.13	1.96	4.07	1.95	2.87	6.10	4.92	9.31	5.99	90.07

summarizes the statistics of the data used for the training of ML-based error models.

Based on the data statistics, it is evident that the type of environment has a significant impact on the accuracy of the position estimations. The vertical accuracy seems to be more influenced by the intensity of the urban environment than the horizontal accuracy is. The accuracy in the urban canyon scenario was about 40 times worse horizontally and about 85 times worse vertically than in the suburban scenario. However, there was no such significant decrease in the average number of visible satellites or increase in the DOP coefficients, even in the urban canyon environments. Furthermore, no instances of unavailability were observed in any multipath-inclusive scenarios.

When considering the impact of multipath signals, we examined two scenarios free from multipath signals in the urban and urban canyon environments. The statistics show that a multipath signal is much more impactful compared to the satellite visibility and geometry in both scenarios. The horizontal and vertical accuracies in the urban scenarios were even better than in the open sky scenarios when the impact of a multipath signal was cancelled, despite slightly increased DOP coefficients and fewer available satellites.

Similarly, in the urban canyon scenario, eliminating the multipath signals also improved the position accuracy. Nevertheless, due to the intense LoS blockage and the high number of NLOS signals in the urban canyon environments, removing all the measurements from the reflected signals led to an availability issue, causing the GNSS availability to decrease to 90.07%. This means that even if the receiver is somewhat capable of detecting and extracting all the faulty measurements from the position estimation process, we cannot guarantee enhancing the accuracy since the availability might be compromised. Therefore, our integrity monitoring approach specialized for urban environments should be able to deal with the presence of reflected signals and possess the ability to calculate PLs accurately.

4.3. Feature Analysis

After analyzing the training data in the previous section, we investigated the training features mentioned in Section 4.1 in order to reveal their impact on the horizontal and vertical error in each scenario. Numerous feature-ranking algorithms are available for regression and classification model training, but one of the most common and efficient algorithms for regression models is Minimum Redundancy Maximum Relevance (MRMR) algorithm [47]. It calculates feature importance scores for each feature, x, based on the Mutual Information Quotient (MIQ) value. MIQ can be defined as,

$${\mathrm{M}\mathrm{I}\mathrm{Q}}_{\mathrm{x}}={\displaystyle \frac{{\mathrm{V}}_{\mathrm{x}}}{{\mathrm{W}}_{\mathrm{x}}}}$$

(20)

where ${\mathrm{V}}_{\mathrm{x}}$ is the relevance, and ${\mathrm{W}}_{\mathrm{x}}$ is the redundancy of features in the response.

The results presented in

Table 2. Feature importance scores (MIQ) of the training features for error estimation.

Features	MIQ for Horizontal Error			MIQ for Vertical Error
	Suburban	Urban	Urban Canyon	Suburban	Urban	Urban Canyon
vS	0.0499	0.0627	0.0554	0.1728	0.1316	0.1686
HDOP	0.0323	0.0829	0.0796	N/A	N/A	N/A
VDOP	N/A	N/A	N/A	0.1690	0.1372	0.3376
${\mathrm{m}\mathrm{i}\mathrm{n}}_{\mathrm{e}\mathrm{l}\mathrm{e}}$	0.0270	0.1083	0.1756	0.1167	0.1515	0.3955
${\mathsf{\mu }}_{\mathrm{e}\mathrm{l}\mathrm{e}}$	0.0462	0.0965	0.0936	0.2479	0.1642	0.3235
${\mathsf{\sigma }}_{\mathrm{e}\mathrm{l}\mathrm{e}}$	0.0476	0.1095	0	0.2542	0.1518	0.4175
${\mathsf{\mu }}_{\mathrm{s}\mathrm{n}\mathrm{r}}$	0.0272	0.1038	0.1369	0.0923	0.1345	0.3850
${\mathsf{\sigma }}_{\mathrm{s}\mathrm{n}\mathrm{r}}$	0.2842	0.3579	0	0.7867	0.5830	0
${\mathsf{\mu }}_{\mathrm{a}\mathrm{z}\mathrm{i}}$	0.3401	0	0.1531	0.9747	0.1799	0.6140
${\mathsf{\sigma }}_{\mathrm{a}\mathrm{z}\mathrm{i}}$	0.0471	0.4338	0	0.2570	0.7088	0.3712
$\left|\right|\mathrm{r}\left|\right|$	0.1109	0.2307	0.3274	0.3258	0.2621	0.4859
${\mathrm{d}}_{\mathrm{m}\mathrm{a}\mathrm{j}\mathrm{o}\mathrm{r}}$	0.0418	0.1003	0.4463	N/A	N/A	N/A
${\mathrm{d}}_{\mathrm{u}}$	N/A	N/A	N/A	0.2883	0.1555	0

provide a comprehensive analysis of the feature importance scores calculated using the MRMR algorithm, focusing on the horizontal and vertical errors across the suburban, urban, and urban canyon environmental scenarios.

For the horizontal error estimation, the MIQ scores varied significantly across the different environmental scenarios. In the suburban environment, the most impactful feature for horizontal error estimation was ${\mathsf{\mu }}_{\mathrm{a}\mathrm{z}\mathrm{i}}$ with an MIQ score of 0.3401, indicating that the mean azimuth angle was the dominant feature in this scenario. ${\mathsf{\sigma }}_{\mathrm{s}\mathrm{n}\mathrm{r}}$ and $\left|\right|\mathrm{r}\left|\right|$ also had significant impacts with MIQ scores of 0.2842 and 0.1109, respectively. The impact of the other features was relatively small.

In the urban environment, the standard deviation of the azimuth and signal strength were the most influential features, with notably high MIQ scores of 0.4338 and 0.3579, respectively. This indicates that the variance-related features are more critical in urban settings, likely due to the complex signal reflections and multipath effects prevalent in such environments. Similar to the suburban scenario, $\left|\right|\mathrm{r}\left|\right|$ remains as one of the important features for the urban scenario as well. The contributions of the other features were still low, but an overall increase in the features’ impact could be observed according to the MIQ scores.

For the urban canyon environment, ${\mathrm{d}}_{\mathrm{m}\mathrm{a}\mathrm{j}\mathrm{o}\mathrm{r}}$ emerged as the most critical feature with an MIQ score of 0.4463. $\left|\right|\mathrm{r}\left|\right|$ and ${\mathrm{m}\mathrm{i}\mathrm{n}}_{\mathrm{e}\mathrm{l}\mathrm{e}}$ also showed a significant influence, with MIQ scores of 0.3274 and 0.1756, respectively. This underscores the importance of the physical dimensions of the urban canyon, which likely affect the signal propagation and, thus, the horizontal error.

For the vertical error estimation, the most impactful features followed a similar trend to that of the horizontal error in the suburban and urban scenarios. This consistency across both dimensions presents the overall impact of these features on the positional accuracy. However, in the urban canyon scenario, the top features for the vertical error estimation were different from those for the horizontal error estimation. ${\mathsf{\mu }}_{\mathrm{a}\mathrm{z}\mathrm{i}}$, $\left|\right|\mathrm{r}\left|\right|$ and ${\mathsf{\sigma }}_{\mathrm{e}\mathrm{l}\mathrm{e}}$ were the most significant features based on their MIQ scores. It is also evident that the norm of LSE residuals, $\left|\right|\mathrm{r}\left|\right|$, is one of the most influential features for all environments in both directions.

Across the different environments, certain features emerged as crucial for error prediction, albeit with varying degrees of importance. This analysis underscores the necessity for environment-specific feature selection in model training to enhance the error prediction accuracy. Understanding the dominant features in each scenario aids in evolving more robust and reliable integrity monitoring solutions to address the unique challenges posed by different urban settings.

With the conventional approaches, there is no possible way to consider these environment-specific impacts because they rely on static error models of open-sky conditions and ignore the dynamic multipath and blockage conditions of urban environments. Therefore, in the next section, we initially investigate the performance of the conventional PL estimation approaches in simulated urban environments to determine the performance differentiations with varying environmental conditions.

4.4. Classical PL vs. IBPL

As a part of this study, the performance of the classical PL and IBPL estimation approaches, which we reviewed earlier, must be compared under simulated urban conditions. This comparison shows us what level of integrity monitoring performance can be achieved and which approach is more suitable for each type of environment. The results are also necessary to validate the proposed NGB-PL approach. While estimating PLs with both approaches, the training data which were analyzed in Section 4.2 have been used. This data set contains 86281 samples with known position errors.

For this comparison, the Stanford-ESA Integrity Diagram [48] is one of the most practical tools to illustrate the integrity events with respect to the position errors (PEs), protection levels (PLs), and alert limits (ALs), as seen in Figure 7.

The events dropping in the white triangular zone, where the PE < the PL < the AL, are considered as the nominal operations. If the PL > the AL, the system is called unavailable because the integrity requirements are not fulfilled. However, if the system is called unavailable but the actual PE is smaller than the AL, it is a false detection. Similarly, if the system is called available where the PL < the AL, but the actual PE is greater than the PL, it is a missed detection and leads to a misleading operation. In reality, missed detections pose a risk to the operation’s safety, while false detections compromise the availability of the navigation system.

Although there is no consensus regarding the required navigation performance in UAM applications, we must decide on the horizontal AL (HAL), vertical AL (VAL), and target IR for integrity analysis. According to ICAO’s signal in space performance requirements for aviation users [49], 40 m and 50 m are suggested for HAL and VAL, respectively, with an integrity risk of less than 1–2 × 10⁻⁷/h for APV-I approach operations. On the other hand, the European GNSS Agency defined more stringent alert limits for safety-critical applications [50] where HAL is 5 to 10 m, and VAL is 10 to 25 m. In between these two suggestions, HAL = 20 m, VAL = 40 m, and the target IR = 1–2 × 10⁻⁷/h values are selected for this study.

The estimated HPL results against PEs are illustrated in Stanford-ESA diagrams for HAL = 20 m in Figure 8, Figure 9 and Figure 10.

According to the test results, both approaches perform best in the suburban test scenarios. However, the IBPL approach performs better than the classical approach, providing 23.5% more availability without causing any misleading information.

With increasing blockage and signal reflections in the urban environment test scenario, the system availability dropped to 11.86% for the classical approach and 2.18% for the IBPL approach. Despite the declining availability, no misleading information was observed with both approaches.

When it came to the urban canyon test scenario, the IBPL approach became totally unavailable as a result of the unnecessarily large PL estimations. On the other hand, the classical approach maintained nominal operation only for 4.36% of the simulation period. The biggest risk in the urban canyon scenario when the classical approach was implemented was the high probability of misleading information with 4.78%.

We can conclude that the overall performance of both approaches is only satisfactory for suburban scenarios. The IBPL approach is superior to the classical approach in the suburban scenario, where faulty measurements are uncommon and satellite visibility is not intensely restricted. However, one approach is fully unavailable for the urban canyon scenario, while the other is not safe to use.

A similar analysis was carried out for the vertical axis as well, and the estimated VPL results against PEs are illustrated in Stanford-ESA diagrams for VAL = 40 m in Figure 11, Figure 12 and Figure 13.

According to the VPL analysis results, again, the IBPL approach is superior to the classical approach in the suburban environment test scenario with a 96.93% nominal operation rate against 0.37%. However, both approaches are mostly unavailable in the urban and urban canyon scenarios except for a 1.52% availability of the IBPL method in the urban test scenario. We can simply conclude that neither the classical nor IBPL approach is able to provide sufficient performance for vertical integrity monitoring in urban and urban canyon environments.

4.5. NGB-PL

In the following subsections, we will assess the performance of the NGB-PL estimation approach using the test and validation data. The results will be compared with PLs estimated via the existing approaches to demonstrate the performance improvements.

For performance assessment and comparison, the following quality metrics will be calculated for each scenario where N represents the number of samples:

• Nominal Operation (NO) Rate = N_NO/N_All;
• Misleading Information (MI) Rate = N_MI/N_All;
• Average Bound Gap (BG) = (PL−PE)/N_All, where the PL > the PE.

Ideally, we expect to minimize the average BG between the PL and PE without causing a significant increase in the MI rates. In this way, we can enhance the GNSS availability in GNSS-challenging urban environments while maintaining the intended integrity requirements.

4.5.1. Test Results

This subsection presents the PL estimation results with randomly selected test data for each environment scenario, separately. Figure 14, Figure 15 and Figure 16 illustrate the horizontal and vertical PE and PL estimations for each test sample, and the performance of PL estimation approaches based on the quality metrics are compared in

Table 3. Comparison of PL estimation approaches in suburban scenario with test data.

Approach	Horizontal			Vertical
	Average BG (m)	MI Rate (%)	NO Rate (%)	Average BG (m)	MI Rate (%)	NO Rate (%)
Classical	18.55	0	75.55	49.93	0	0.47
IBPL	4.83	0	97.99	11.81	0	97.06
NGB-PL	0.65	0.01	99.99	0.89	0.02	99.98

,

Table 4. Comparison of PL estimation approaches in urban scenario with test data.

Approach	Horizontal			Vertical
	Average BG (m)	MI Rate (%)	NO Rate (%)	Average BG (m)	MI Rate (%)	NO Rate (%)
Classical	21.03	0	11.42	50.37	0	0
IBPL	90.45	0	2.23	201.36	0	1.58
NGB-PL	5.09	0	100	15.58	0	97.67

and

Table 5. Comparison of PL estimation approaches in urban canyon scenario with test data.

Approach	Horizontal			Vertical
	Average BG (m)	MI Rate (%)	NO Rate (%)	Average BG (m)	MI Rate (%)	NO Rate (%)
Classical	19.67	4.75	4.14	34.79	10.61	0
IBPL	594.25	0	0	1122.05	0	0
NGB-PL	28.69	0	4.69	67.71	4.28	0.06

.

According to the test results, one of the key outcomes of the tests is that the NGB-PL approach enhances the GNSS availability by increasing the NO rates for each environment type in both the horizontal and vertical axes. This is, in fact, because of narrowing the average BG. However, narrower bound gaps can cause additional integrity risks due to a rising MI rate. This situation could be observed in the urban canyon scenario where the classical approach provided a narrower average BG, but the MI rate reached 4.75% for the horizontal axis and 10.61% for the vertical axis. Additionally, despite the increased risks, no increase in availability could be observed with the classical approach in the urban canyon scenario.

As expected, all the approaches performed best in the suburban environment due to the limited signal blockage and reflection. The NGB-PL approach is superior to the other two approaches even in this less challenging environment where the horizontal and vertical NO rates were improved by 2% and 2.92%, respectively. Although an increment in the MI rates of 0.01% in the horizontal and 0.02% in the vertical axes were observed, it did not lead to any integrity risks since the actual positioning errors did not exceed the ALs.

The most considerable enhancement in the availability was observed in the urban scenarios utilizing the NGB-PL approach. GNSS became 88.58% more available for the horizontal and 96.09% more available for the vertical position estimations without leading any MI.

4.5.2. Validation Results

In this subsection, the PL estimation results with continuous validation data are presented separately for each environment scenario. Figure 17, Figure 18 and Figure 19 illustrate the horizontal and vertical PE and PL estimations for each validation sample, and the performance of the PL estimation approaches based on the quality metrics are compared in

Table 6. Comparison of PL estimation approaches in suburban scenario with validation data.

Approach	Horizontal			Vertical
	Average BG (m)	MI Rate (%)	NO Rate (%)	Average BG (m)	MI Rate (%)	NO Rate (%)
Classical	19.67	0	51.42	50.88	0	0
IBPL	1.53	0	100	3.84	0	100
NGB-PL	0.37	0.89	99.11	0.95	0	100

,

Table 7. Comparison of PL estimation approaches in urban scenario with validation data.

Approach	Horizontal			Vertical
	Average BG (m)	MI Rate (%)	NO Rate (%)	Average BG (m)	MI Rate (%)	NO Rate (%)
Classical	21.46	0	0	56.53	0	0
IBPL	103.44	0	9.11	227.49	0	7.31
NGB-PL	6.05	0	100	12.08	0	100

and

Table 8. Comparison of PL estimation approaches in urban canyon scenario with validation data.

Approach	Horizontal			Vertical
	Average BG (m)	MI Rate (%)	NO Rate (%)	Average BG (m)	MI Rate (%)	NO Rate (%)
Classical	19.65	1.41	0	33.68	2.16	0
IBPL	281.83	0	0	545.70	0	0
NGB-PL	25.64	0.05	5.61	58.07	0	0

.

According to the validation results, in suburban environments, the NGB-PL and IBPL approaches offer similar performance with respect to the NO rate, but IBPL is slightly better than NGB-PL in the horizontal axis. Nevertheless, NGB-PL provides a narrower average BG, which is beneficial for missions in more restricted airspace among the buildings.

In the urban scenario, similar to the test results, the proposed NGB-PL approach performed much better than the other approaches with validation data and the increasing availability of GNSS navigation by more than 90% both horizontally and vertically.

Lastly, in the urban canyon scenario, the NGB-PL approach was capable of providing 5.61% GNSS availability with a narrower average BG, and the MI rate was decreased by 1.36% compared to the classical approach. However, neither the conventional nor the proposed approaches can offer an intended level of availability for vertical positioning.

5. Discussion

The results of this study provide significant insights into the GNSS integrity monitoring in the context of urban air mobility (UAM), specifically in suburban, urban, and urban canyon scenarios. Within the scope of the study, three models for urban environments were proposed based on the signal blockage and reflection intensity. These environment models were initially utilized in our simulation setup to conduct an environment-specific analysis of the GNSS performance. This analysis demonstrated how the GNSS position accuracy, availability, and satellite visibility/geometry were impacted by different types of environments.

In the multipath-inclusive scenarios, despite there being no availability issues or significant decrease in the average number of visible satellites, the accuracy in the urban environment became 8 times worse horizontally and 18 times worse vertically, and the accuracy in the urban canyon environment became 40 times worse horizontally and 85 times worse vertically compared to the suburban environment. On the other hand, in the multipath-free scenarios where the reflected signals were removed, it was observed that the horizontal and vertical accuracy in the urban environment was as good as the accuracy in the open sky scenario. However, in the urban canyon scenario, although the horizontal accuracy was 6.5 times better and the vertical accuracy was 12.5 times better than in the multipath-inclusive scenario, the GNSS availability decreased to 90% because of the insufficient healthy signals. These results showed that the integrity monitoring with MRA is not a solution for dense urban areas. Therefore, we decided to enhance the integrity monitoring with ECA without eliminating all the faulty measurements.

Since the integrity monitoring performance with ECA relies on accurate PL estimations, generating environment-specific error models is necessary considering the dynamic nature of urban settings. Therefore, a feature analysis was conducted to reveal the importance of the GNSS and environment-based features while training error models for each type of environment. The feature importance analysis results demonstrate that azimuth-related features have the most impact on suburban and urban error models. Additionally, ${\mathsf{\sigma }}_{\mathrm{s}\mathrm{n}\mathrm{r}}$ and $\left|\right|\mathrm{r}\left|\right|$ are among the top features in suburban and urban scenarios both horizontally and vertically. $\left|\right|\mathrm{r}\left|\right|$ also remains one of the most crucial features for horizontal and vertical error estimation in the urban canyon scenario. While ${\mathrm{m}\mathrm{i}\mathrm{n}}_{\mathrm{e}\mathrm{l}\mathrm{e}}$ and ${\mathrm{d}}_{\mathrm{m}\mathrm{a}\mathrm{j}\mathrm{o}\mathrm{r}}$ are the other most impactful features of the horizontal urban canyon error model, ${\mathsf{\mu }}_{\mathrm{a}\mathrm{z}\mathrm{i}}$ and ${\mathsf{\sigma }}_{\mathrm{e}\mathrm{l}\mathrm{e}}$ have more influence on the vertical error model. Although we benefited from all the available features during the model training in this study, this feature analysis would be beneficial for feature selection where the computational power is limited.

The performance of the proposed integrity monitoring framework with the NGB-PL approach was investigated after data and feature analysis. Initially, classical PL and IBPL estimations were compared with each other using the training data set in order to demonstrate their strengths and weaknesses in different sorts of environments. Then, NGB-PLs were estimated with test and validation data in addition to the PLs estimated with the classical and isotropy-based approaches.

If we consider the environment types one by one, it was observed that in the suburban scenario, where the blockages and reflections were negligible, the NGB-PL and IBPL approaches performed efficiently with above 97% availability both horizontally and vertically. Moreover, NGB-PL significantly reduced the average BG. With a narrower average BG, our proposed approach can maintain higher GNSS availability where the required navigation performance is stricter. The classical PL estimation approach, on the other hand, maintained very large bound gaps between the PL and PE in the suburban scenario and caused unnecessarily high unavailability. It would be a safe approach for aviation use cases, but such an over-conservative approach is impractical for UAM applications.

In the urban environment scenario, the NGB-PL approach was significantly more efficient than the other methods, simultaneously providing a balanced performance with a narrower bound gap and higher availability. The GNSS became approximately 90% more available with the NGB-PL approach. Additionally, no integrity risk was observed, neither in the horizontal nor vertical positioning. Unlike the suburban scenario, the IBPL approach is fragile against faulty measurements in urban environments and reacts dramatically to even minimal position error changes, as seen in Figure 18. Although the classical approach provides lower bound gaps than IBPL, IBPLs are still more compatible with UAM integrity requirements to maintain availability and avoid MI.

Lastly, the urban canyon scenario was challenging for all the PL estimation methods due to the increased number of faulty measurements. Even the actual position errors were significantly higher in the urban canyon scenario compared to the others. The NGB-PL approach can still maintain limited availability while not leading to high integrity risks. However, the results show that GNSS-only integrity monitoring is inefficient via both the conventional and proposed methods for such highly intense urban environments.

Overall, according to the results, the error and PL estimation accuracy are highly dependent on the environment type, and the proposed framework with additional environment recognition functionality would be beneficial for the following reasons:

• With the development of environment-specific error models utilizing selective features, more accurate error predictions and corrections can be applied within specific types of environments.
• Lower average bound gaps and more GNSS availability can be provided without leading to additional integrity risks.
• Users can avoid specific environment types to maintain the required GNSS integrity and availability.
• Since the performance of the integrity monitoring methods varies with the changing environmental conditions, a switching algorithm can be implemented to assign a particular approach to a particular environment type.

6. Conclusions

In this study, we developed a novel ML-based GNSS integrity monitoring framework in order to address the unique challenges of urban environments by incorporating environment-specific error models. The proposed framework, which integrates the NGB algorithm, demonstrates substantial improvements over classical PL and IBPL estimation approaches in various urban scenarios.

Our results showed that the NGB-PL approach significantly enhances the performance, particularly in urban and suburban environments, by reducing the bound gap and increasing the GNSS availability. Unlike other PL estimation methods, which often lead to unnecessarily high unavailability, the NGB-PL method maintains narrower PLs, compromising integrity and making it more suitable for UAM applications.

The integration of environment recognition into the integrity monitoring framework allows for the development of environment-specific error models. This enhancement enables more accurate error predictions and integrity monitoring in specific environment types. Moreover, it allows users to avoid certain environment types to maintain the required integrity and availability and switch between different integrity methods based on the environment.

Overall, the findings underscore the importance of considering environment-specific factors in GNSS integrity monitoring. The proposed NGB-PL approach, with its environment recognition capabilities, offers a promising solution to enhance the reliability and safety of GNSS-based navigation systems in UAM applications.

Future work should explore extending this framework to additional scenarios and environment types and integrating additional environmental features to further improve the accuracy and availability in more challenging UAM applications. The proposed approach should also be validated with an experimental setup after more diverse environment models have been developed in addition to those provided in this study. In the proposed framework, we assumed that FDE and environment recognition units are seamless in order to measure the true potential of the NGB-PL approach. However, the existing methods in the literature for environment recognition are not able to provide sufficient performance to detect environment types in such demanding UAM scenarios. The development of the Environment Recognition Unit is outside the scope of this study but should be considered for future works. According to our literature review, Natural Gradient Boosting was chosen as a probabilistic regression algorithm due to its superior performance, but researchers might be interested in investigating other alternatives and comparing them with NGB.