4.1. Main Experimental Results
A comparative analysis was conducted to evaluate the performance of various machine learning algorithms in diagnosing blockage faults within the sensing lines of pressure transmitters. The selected algorithms included SVM, k-Nearest Neighbors, Logistic Regression, Random Forest, ANN, and CNN [
23,
24,
25,
26]. The dataset collected from the experimental platform (
Figure 3) was divided into training and testing sets at an 8:2 ratio. Each model’s performance was assessed and compared via metrics such as accuracy, recall, F1 score, and precision. The performance of these models on the test set is displayed in
Figure 8. The results indicate that the XGBoost algorithm achieves the highest scores across all the evaluation metrics, notably attaining a test accuracy of 99.39%. The test accuracies of all the algorithms are presented in
Table 5. To verify the robustness of our model on small-scale datasets and its applicability in scenarios with limited data, we conducted a series of comparative experiments. The results of these experiments are detailed in
Appendix A. Furthermore, in
Appendix B, we discuss the sensitivity of the proposed method to noise.
In this study, the aforementioned algorithms were tested to assess the robustness of various models in handling the sensing line dataset. The test accuracy of these models in distinguishing between the normal and faulty states of the sensing line is presented in
Figure 9a. In this figure, 0 and 1 represent the two working states of the sensing line, corresponding to normal and faulty conditions, respectively. Furthermore, the variance in test accuracy for each category of data across the seven algorithms is depicted in
Figure 9b.
As shown in
Figure 9a, the test accuracy of the XGBoost algorithm exceeds 99% for datasets in both normal and faulty states. This exceptionally high accuracy indicates the reliability of the XGBoost algorithm in handling such problems. Furthermore, as observed in
Figure 9b, the variance in test accuracy for this algorithm across both dataset types is only 0.0000025, confirming the model’s robustness.
The prediction results of XGBoost and six comparative algorithms on the test set are presented as confusion matrices in
Figure 10. In these matrices, the
x-axis represents the predicted labels, whereas the
y-axis represents the true labels. The value at position (i, j) in the matrix indicates the number of samples with a true label of
j that were predicted as
i. The confusion matrix not only identifies the sample distribution across different categories in the test set but also provides a visual evaluation of the model’s predictive capabilities.
Figure 10 shows that the XGBoost model achieves a prediction accuracy of 99%, further confirming its outstanding performance. In addition, the ROC (Receiver Operating Characteristic) curves, which are widely used for evaluating classification models, are displayed in
Figure 11. Among the ROC curves of the seven algorithms, XGBoost has an AUC (Area Under the Curve) approaching 1, significantly outperforming the other six algorithms. In
Figure 12, in the boxplots of the seven algorithms, the performance of XGBoost is clearly due to the other six algorithms. This demonstrates the superior classification ability and robustness of XGBoost, making it particularly effective in the context of this study.
By comparing the performance of multiple algorithms across four key metrics—accuracy, precision, recall, and F1 score—the superiority of our proposed method is clearly demonstrated. The results reveal that the mean performance of our method across all the metrics is close to 1.0, significantly outperforming the other algorithms. Furthermore, the boxplot results show that the performance variation of our model is nearly negligible, highlighting its superior stability compared with other algorithms. This demonstrates that our method not only achieves a leading position in overall performance but also maintains exceptional robustness under varying conditions, such as changes in random seeds and data splits.
In contrast, Random Forest, as the second-best traditional machine learning algorithm, performs relatively well across all four metrics. However, its mean precision, recall, and F1 score remain slightly lower than those of our model, with marginally greater performance variation. Additionally, deep learning models, such as CNN and ANN, exhibit some instability, particularly ANN, which show significant fluctuations in recall and F1 scores, with some runs yielding low results. This instability reduces their reliability in practical applications. Traditional algorithms, such as SVM and Logistic Regression, perform considerably worse than our method does, with mean accuracy and precision scores below 0.65, indicating their limited suitability for the current task.
The p values for all four metrics are far below 0.05, confirming that the performance distributions of the different algorithms are significantly different. These differences are not due to random errors but are meaningful and statistically significant.
Traditional methods (SVM, KNN, Logistic Regression, Random Forest) rely on statistical features directly extracted from raw data, which fail to adequately capture trend information and time-frequency characteristics. In contrast, the TFTF method extracts both time-domain and frequency-domain features from trend components through multi-scale decomposition, significantly improving classification performance. While SVM and Logistic Regression perform well on linear data, they struggle with the nonlinear and complex distributions commonly found in industrial datasets. XGBoost, with its splitting strategy and weighted voting mechanism, handles nonlinear relationships more effectively, resulting in higher accuracy. ANN and CNN typically require a large amount of training data, which is often limited in industrial scenarios. XGBoost has a lower dependency on the data volume and efficiently utilizes the extracted features. While CNN automatically extract features, these features rely primarily on spatial structures and are less effective in capturing trend information in the time-frequency domain. The TFTF method, which involves pre-extracting specific features and leveraging XGBoost for classification, has clear advantages.
The decision tree structure in XGBoost naturally captures nonlinear relationships and trends within the data. By selecting optimal split points and leaf nodes, the decision tree can identify and learn the trend patterns within the data, allowing it to incorporate trend information into predictions. The regularization term in XGBoost effectively controls model complexity and mitigates overfitting. When handling trend data, proper regularization helps the model avoid overfitting noise while fitting the trend, thereby improving prediction accuracy.