Terminal Traffic Situation Prediction Model under the Influence of Weather Based on Deep Learning Approaches
Round 1
Reviewer 1 Report
The paper describe a machine learning approach for prediction of air traffic situation. Overall, the paper does not have much to add too much to the existing scientific knowledge in ML based air traffic prediction. Some of the choices in the design are quite unclear. My comments are as follows:
1. It is not clear if weather feature is use in the classification step to determine the traffic situation. The authors should make it clear that the weather information should be excluded from this step (and I hope it is so).
2. The use of "good", "average", and "bad" to quantify the traffic situation is an over-simplified. I don't see this been a very useful information in ATC in general, since much of this can already be easily seen based on traffic demand information with fairly straight forward, and deterministic approaches
3. The construction of scenario 3 is unclear. You mush specify what criteria are used.
4. In addition, I would really like to see information on the importance of different features based on your PCA analysis.
5. As indicated by results in 4.2.1, the actual improvement using weather only improves the prediction by about 1%. This is a very low significance.
6. The figures showing the results in section 4 are very confusing, and wrong. The lowest value should start from 0, not some arbitrary values. Such kind of visualizations clearly distort the results.
7. In fact, please avoid such visualizations in the future, instead, you can simply use tables for the error metrics. It will be easier for the readers to compare.
8. In the later section of the paper, the authors start to add many different ML models, including CNN, RNN, LSTM, GRU, etc. This is very confusing. I would stick to just one model that works the best without the weather features. And see what the added value of weather features.
9. Figure 10 and 11 need to be re-designed. I am not able to understand these.
10. The conclusion/discussion section lack of concrete quantitative descriptions on the performance of the research.
11. The organization of the paper sections is also not clear. For example, "solution" section is just the method, "experiment" section only describes the results. I also don't see in-depth discussion beyond the apparent results on why certain approaches works better than others.
12. Finally, none of the code or data related to the research is open, so there is no way for readers to validate the proposed approach. I think this is a major shortcoming.
Author Response
Please see our responses in the attachment.
Author Response File: 
Author Response.pdf
Reviewer 2 Report
The prediction of terminal traffic situations presents an interesting application of Deep Learning Approaches to aerospace applications. The paper is easy to read and gives a good overview of related research in the introduction.
Some parts are not so easy to understand maybe resulting from incorrect references to tables and literature and incomplete sentences (cp. e.g. lines 106/107, 115/116, 120/121, 122-124, 241/242, 241). From line 105 onwards in section 2, authors mentioned in references within the text often do not match to the literature in the ‘References’ section. In Fig. 1, MATAR is part of the data of the weather feature, shouldn’t this be METAR?
In addition, I have some more detailed comments regarding the content:
- p.5, l 201 – 205: What do you mean by ‘waypoint interval’?
- p.6, line 233: Table 2 includes weather scoring standards not the terminal area feature set. There is no table depicting the three categories weather, traffic demand and strategy.
- Rephrase sentence in line 236/237. Do you mean that scenario 2 clusters the traffic situation based on the traffic demand only? The description and table 3 seem to be different.
- Explain why you have chosen the silhouette co-efficient method to determine the number of clusters. There are abundant measures available for this task. Refer to literature explaining the selection of an appropriate number of clusters.
- Table 3: Without information on data covered by the clusters (e.g. min/max values, variance) the presentation of values for the cluster representatives gives no further insights.
- Figure 3: Explain what is depicted in the figure. What do the axes represent (e.g. pca_1, tolerance)?
- Line 299: Correct the setting of symbols.
- Line359/360: What values of R-square do you expect for a representative model? Do your experiments reach those levels?
- Figures 6 to 9: The same information is depicted on the left and the right side of the figures. Omit the images on the right in Figures 6-8 and those on the left in Figure 9. Use the space to elaborate differences in e.g. R-Square (depending on the scenario in Figure 7) in more detail and describe the use of real-data for the different time horizons.
- Figure 10: State clearly what is depicted in the figure. What do the numbers on the axes mean? Which input data is used for which time horizon (e.g. if predicting for 6 hours in advance there can be only weather forecasts with the 6-hour horizon used, no real (weather) data from post-analyses for the prediction period)?
Figure 11: The number is used twice.
Figure 11-1 (p.18): At which point in time starts the forecast / up to which point in time is real data used as input?
Figure 11-2 (p.19): Use more timestamps on the x-axis. No visibility constraints between 9.30 and 13.00 are depicted in the Figure (cp. line 440). What is meant by ‘Label’ and ‘Average Interval’?
Page 20, line 485: Does prediction performance refer to R-Square or computational performance? What about computation times of the models are they comparable?
Author Response
Please see the attachment.
Author Response File: Author Response.pdf
Reviewer 3 Report
The authors propose a mixed deep learning/fuzzy logic approach to implement a model for terminal traffic prediction at intervals of 1, 3 and 6 hours. Intelligent air traffic management is a trend topic in civil aviation, and the paper could potentially add knowledge to the field with the proposed methodology. The introduction defines well the problem, and the bibliographical research describe adequately the work by other research groups. Nonetheless, the paper needs a thorough revision of English style and grammar. There are many sentences grammatically incorrect (e.g. lines 61-64 (missing verbs), lines 106-107, lines 115-116, lines 120-121, lines 122-124 (probably “Obtain” at the beginning of the sentence should be “obtaining”, and the sentence should be joined to the preceding one with a comma), lines 95-96 (“Determine” should read “determining” and a comma should join the two sentences), lines 241-242, lines 261-263, lines 326-337 (useless repetitions)), together with some typos (line 126, “assessment.” Should read “assessment,”; line 141, “Maria [33] et al.” should read “Prandini et al. [33]”; line 231, “scenarios” should read “scenario”; Figure 3 (what are pca_1, pca_2 and pca_3 in the leftmost diagram?); line 207, “an” should read “a”; line 274, “Where: is” should read “where Ol (O with subscript “l”); Table 2 (for the sake of readability, horizontal lines should divide the text for the six weather classes); Heading of Sec. 4 (“Experimental” should read “Experimental results”).
Furthermore, the methodologies are somewhat confusing (or at least poorly explained, for example the TAWQM method). As far as the results are concerned, it seems that the improvement with respect to “predictions without strategy features” does not seem so dramatic (less than 3% average, in few cases greater than 10%). Moreover, the predictions relative to “scenario 3 (T=6)” are not so outstanding, as shown in Figure 10(i). What is the computational cost of the proposed methodology? What is the actual advantage of using the approach proposed by the authors with respect to the “traditional” methods? Why did the authors choose a fuzzy c-means clustering instead of "traditional" clustering methodologies?
Based on these comments, I think that the paper is not publishable in its current form and it needs a deep revision before publication.
Author Response
Please see the attachment.
Author Response File: Author Response.pdf
Round 2
Reviewer 1 Report
Most of my comments are addressed. Here are the remaining ones:
"Point 4: In addition, I would really like to see information on the importance of different features based on your PCA analysis.
Response 4: Thank you for your comment.PCA is a method to achieve data dimensionality reduction by mapping high-dimensional features to a low-dimensional space. The principal components obtained after mapping are no longer a part of the original features, but the combined calculation 2 result of the original features, which is uninterpretable. Therefore, we may not be able to explain the importance of the original features with PCA results in the text. I hope this response will satisfy you."
I am not satisfied with this response. The output of the dimension reduction are the most significant features for the predictor, and that's what PCA does. I would still like to see, among all the features you selected, which ones have the most impact for your predictions.
"Point 9: Figure 10 and 11 need to be re-designed. I am not able to understand these.
Response 9: Thank you very much for your comment. In the revised manuscript, we have modified the figures in Section 4 into tables for easier reading. Hope the revised version will satisfy you."
I am referring to Figure 4 in the update manuscript, which was the figure 11 on page 19 of the old version. This figure is not possible to read. There are 1) secondary axes which are hard to understand, 2) duplicated line style and color coding, 3) colors (e.g. yellow) that are nowhere in the figure, and 4) histogram for label (?). You must improve this figure. I am not able to read useful information/conclusion from this figure.
"Point 12: Finally, none of the code or data related to the research is open, so there is no way for readers to validate the proposed approach. I think this is a major shortcoming.
Response 12: Thank you very much for your comment. If necessary, we can make the data and code of this work publicly available."
Yes, please do so. I would like to see them being shared in a repository (like figshare, or zenodo), and with the DOI included in this paper.
Author Response
Please see the attachment.
Author Response File: Author Response.pdf
Reviewer 2 Report
Thank you for the improvement of the paper.
A few typing errors remained in the paper:
- Table 2, 4): degree symbol not correctly set
- Table 11: write 7th instead of 7rd in table heading
Author Response
Response to Reviewer 2:
Point 1: - Table 2, 4): degree symbol not correctly set
Response 1: Thanks for your comment. We have modified the symbol settings in Table 2.
Point 2: - Table 11: write 7th instead of 7rd in table heading
Response 2: Thanks for your comment. We have revised the table headings in Table 11.
Hope the revised version will satisfy you.
Author Response File: Author Response.pdf
Reviewer 3 Report
I am satisfied with the revisions made by the authors. There are still some corrections to be made. In lines 115-116 still survives the sentence “For the study of classification model prediction.” Perhaps the sentence should be terminated with a comma, referring to the work of Hoffmann et al. [14-18]. The same observation stands for line 121 (“For deep learning methods prediction.” Should have a comma instead of a period). In Figure 1, “featrue” should read “feature”. The sentence in lines 273-276 is not clear. There is a question mark in Eq. (10).
After these minor corrections, I would recommend publication.
Author Response
Response to Reviewer 3:
Point 1: There are still some corrections to be made. In lines 115-116 still survives the sentence “For the study of classification model prediction.” Perhaps the sentence should be terminated with a comma, referring to the work of Hoffmann et al. [14-18]. The same observation stands for line 121 (“For deep learning methods prediction.” Should have a comma instead of a period). In Figure 1, “featrue” should read “feature”. The sentence in lines 273-276 is not clear. There is a question mark in Eq. (10).
Response 1: Thank you very much for your comment. Based on your suggestion, we have reread the paper several times and corrected mistakes. Mistakes and incomplete sentences in the formula have been corrected, and the errors in Figure 1 have been corrected. Hope the revised version will satisfy you.
Author Response File: Author Response.pdf
Round 3
Reviewer 1 Report
The authors have sufficiently addressed my comments.
