In this section, two types of experiments are conducted to evaluate the performance of the PHFET on classification and capability demand satisfaction degree evaluation. To achieve this, we utilize data sets obtained from the UCI machine learning repository, alongside a case simulation of SCS.
5.1. Verification on Classification
The applications in this study utilize several data sets sourced from the UCI machine learning database. These data sets include the Statlog (Australian Credit Approval) data set, Breast Cancer Wisconsin (Diagnostic) data set, Seeds data set, Climate Model Simulation Crashes data set (CMSC), Heart disease data set, Wine data set, and Ionosphere data set. The details are shown in
Table 1.
To demonstrate the effectiveness of the proposed method in classification, several algorithms are selected. These algorithms include XGBoost, Support Vector Machine (SVM), Random Forest (RF), Neural Network (NN), and Logistic Regression (LR). For each dataset, we have chosen ten classifiers from these five methods to generate a diverse set of classifiers. The specific classifiers used are as follows:
XGBoost classifiers with booster options of gbtree and gblinear.
SVM classifiers with radial basis function kernel and linear kernel.
RF classifiers with criterion options of gini and entropy.
Multi-Layer Perceptron (MLP) classifiers with two hidden layers using either 10-10 or 20-10 nodes, and tansig activation function.
LR classifiers with LBFGS and Stochastic Average Gradient (SAG) solver, respectively.
To combine the classifiers, the PHFET method is used. Five types of classifiers are utilized, resulting in five bodies of evidence. Each body of evidence includes information from two classifiers. The mass function of evidence is generated using the output of two classifiers and their accuracy, according to Equations (
34) and (
35). In the experiments, the 5-fold cross-validation method is employed, which is a common method to test the accuracy of the classification algorithm. The mean accuracy of different classifiers based on different data sets is provided in
Table 2.
As observed from
Table 2, the PHFET method consistently achieves the highest accuracy across all seven data sets. These results clearly demonstrate the effectiveness of the PHFET method in merging the information and advantages of multiple classifiers, leading to improved recognition accuracy.
5.2. Application on Capability Demand Satisfaction Evaluation of SCS
To demonstrate the effectiveness of a practical SCS capability evaluation, a specific information assurance mission is considered. This evaluation aims to assess the SCS’s ability to meet the requirements of the mission task. The SCS consists of the walker constellation, with a total of 24 satellites, 4 orbital planes, and 6 satellites per orbital plane.
The objective of this operation is to safeguard maritime and land-based communications, which can be further divided into three tasks: command communications, reconnaissance intelligence transmission, and daily communications tasks. These activities involve issuing command orders, transmitting and receiving positioning information and weather updates, facilitating daily communications, and providing broadcasting services. By analyzing each activity individually, a list of SCS capability demands under the information assurance mission is compiled, along with corresponding capability indicators. These indicators include ground coverage, orbital coverage, time coverage, inter-satellite link connectivity, signal-to-noise ratio, bit error ratio, link interruption rate, packet loss ratio, bandwidth, time delay, transmission rate, throughput capacity, denoted as to .
The evaluation considers three concentrations within the FoD:
where
represent the satisfaction levels of high, medium, and low, and the boundaries of
, and
are not precisely defined.
Assume that there are two potential schemes for constructing the SCS to fulfill the mission. The demand indicator values of different satisfaction levels have been provided by experts, and the indicator values of different schemes are collected from simulation. The values of the demand indicators for maritime communication are , , , , , , , , ,, , times higher than that for land, respectively.
In order to integrate the different indicator values, the raw data need to be standardized and converted into normalized data with a range of
. The demand indicators of the land communication assurance mission and the indicators of two schemes are given in
Table 3.
The importance of two parts of the mission under 12 indicators is shown in
Table 4.
According to the distance-based generation method, the data of 12 indicators are modeled as PHFBPAs, which consist of the importance and the degree of affiliations of demand of two parts of the mission. The detailed PHFBPAs of scheme 1 are shown in
Table 5.
As depicted in
Table 5, an inconsistency arises between
and other evidence since
assigns more belief mass to satisfaction level
compared to
, which is supported by other evidence. To ensure a comprehensive synthesis, it is crucial to merge the various pieces of data. Relying solely on a single piece of evidence would be unreliable for making informed decisions. Therefore, in order to address the uncertainties and conflicts, the reliability discounting factor is determined by employing the Jousselme distance of PHFBPAs, while the credibility discounting factor is determined using the entropy measure of PHFBPAs.
According to Equation (
37), the credibility discounting factors could be calculated as follows:
Following Equation (
42), the reliability discounting factors could be calculated as:
The credibility discounting factors and the reliability discounting factor are integrated to form the final weight to adjust the PHFBPAs of the evidence. Applying the combination rule of PHFET Equation (
18) to fuse the modified evidence 11 times and use the score function of PHFBPA to obtain the final result as:
. These scores indicate a high degree of satisfaction with the capability demand for scheme 1, suggesting that the Satellite Communication System (SCS) built according to this scheme possesses the necessary capabilities to successfully fulfill the mission.
To facilitate a comparison between different schemes, assume that there are scheme 3 and scheme 4. Scheme 3 shares identical indicators with scheme 1, except for
and
. On the other hand, scheme 4 has identical indicators to scheme 1, except for
. The fusion result for these four schemes are illustrated in
Figure 4.
Analyzing the satisfaction degrees of the four schemes reveals that scheme 1, scheme 3, and scheme 4 exhibit high levels of satisfaction, while scheme 2 demonstrates a moderate level. The discrepancies in the indicator values account for the variations in satisfaction levels among scheme 1, scheme 3, and scheme 4. Notably, scheme 3 yields the highest level of satisfaction. However, despite a notable reduction in packet loss ratio compared to scheme 1, its impact on overall improvement is minimal.
5.3. Discussion
To assess and validate the stability of the proposed algorithm, we conducted sensitivity analysis on the indicator weights and mission importance to examine their impact on the fusion result. We assigned different weights to the 12 indicators, denoted as
in Equation (
43), creating three distinct weight sets, as shown in
Table 6.
Furthermore, in order to evaluate the effect of the basis for the possibility values in PHFBPAs on the fusion result,
Table 7 presents two more importance ratings for land and maritime communication missions in addition to those in
Table 4.
The experimental results, depicted in
Figure 5, reveal that the priority order of satisfaction level remains consistent despite significant fluctuations in both indicator weights (
Figure 5a) and mission importance (
Figure 5b). These findings strongly support the stability and robustness of the introduced PHFET model under various weighting scenarios.
These results reaffirm the effectiveness and reliability of the proposed algorithm for the decision-making processes. The algorithm’s ability to maintain consistent performance across different weight configurations enhances its practical applicability. Decision-makers can confidently use this model without concerns about unpredictable or inconsistent outcomes due to variations in experts’ weights. Additionally, the stability analysis provides a solid foundation for future research and potential refinements of the algorithm.
Several traditional methods have been adopted for comparison, including Dempster’s method [
39], referred to as ‘DS’; Yager’s method [
40], referred to as ‘Yager’; Sun et al.’s method [
41], referred to as ‘Sun’; Murphy’s method [
42], referred to as ‘Murphy’; and Deng’s method [
36], referred to as ‘Deng’. To verify the effectiveness of the proposed discounting factors in eliminating uncertainty and conflicting evidence, several variants of the PHFET method are utilized. The PHFET method without the discounting of evidence is denoted as ‘PHFET’, while the versions with credibility discounting factor and reliability discounting factor are denoted as ‘PHFET-1’ and ‘PHFET-2’, respectively. Furthermore, the combination of both discounting factors is denoted as ‘PHFET-12’.
According to the distance-based generation method proposed in the previous section, PHFBPAs of scheme 1 were obtained. Furthermore, to facilitate comparison with other methods, the adjustment parameter
in Equation (
31) takes the values of 1 and 1, with a probability of
for each value. The final fusion results of the evidence from all indicators are depicted in
Figure 6.
As can be seen from
Figure 6, most of the methods allocate the largest belief mass to
, indicating sufficient capabilities to carry out the mission as intended, except for Yager’s method and Sun et al.’s method. These two methods allocate most of the belief mass to an unknown space
V, indicating that they cannot provide a specific satisfaction level. Among the methods that identify the satisfaction level as high, the discounted PHFET method performs the best, achieving the highest belief of
and demonstrating superior convergence performance by quickly converging to 1. Additionally, compared to the PHFET method without discounting, which assigned the belief mass of
to
, and only uses one of the credibility and reliability discounting factors, which results in belief masses of
and
, respectively, the PHFET method with both discounting factors allocates a higher belief degree to the target concentration. Thus, the effectiveness and superiority of the uncertainty and conflict-based discounting strategy of PHFET is demonstrated.