Comparison of the Forecast Accuracy of Total Electron Content for Bidirectional and Temporal Convolutional Neural Networks in European Region
Round 1
Reviewer 1 Report
- The meaning of the acronyms GRU (Gated Recurrent Unit) must be given in line 41 instead of line 113.
- The meaning of the acronyms LSTM (long short-term memory), TCN (Temporal Convolutional Networks), CODE (Center for Orbit Determination in Europe), ANN (Artificial Neural Network), IGS (International GNSS Service), MAE (Mean Absolute Error), GIM (Global Ionosphere Map), SSN (Solar Sunspot Number), MAPE (Mean Absolute Percentage Error), RRN (Recurrent Neural Network), UPC, ED-LSTME, cGAN, JPL, must be written explicitly in the paper.
- In order to make fluent the paper I strongly recommend the following modifications:
a) “An example of using the LSTM architecture for TEC prediction is the paper [8], which…” replace with: “An example of using the LSTM architecture for TEC prediction is the paper which….”
b) “In [9], the authors developed and applied ….” replace with: “Sun et al. [9] developed and applied ..”
c) “In [10], the BiLSTM algorithm was applied” replace with:”Sivakrishna et al.. [10] applied the BiLSTM algorithm to forecast one hour ahead the Indian….”
d) “In [11], several architectures ……are presented in Appendix of paper [11]” replace with: “ In Chen et al [11], several architectures…… are presented in the appendix….”
e) “In [12], TEC forecasting model based on deep …” replace with: “ A TEC forecasting model based on deep learning was proposed by Tang et al. [12], it consists….”
f) “Another conventional model – Gated Recurrent Unit (GRU) was compared in [13] with LSTM model results from equatorial station MAL2 data in Kenya and was more accurate than LSTM and other methods (Multilayer Perceptron (MLP), GIM_TEC and the IRI-Plas 2017) “ replace with: Iluore and Lu [13] analysing data from the equatorial station MAL2 (Kenya) found that GRU was more accurate than LSTM, Multilayer Perceptron (MLP), GIM_TEC and the IRI-Plas 2017.
g) “In [14] a spatiotemporal deep learning architecture CNN-GRU combining a convolution neural network (CNN) to capture the spatial variability of TEC, and a gated recurrent unit (GRU) for temporal variability modeling has been proposed.” replace with:” Kaselimi et al. [154] proposed a spatiotemporal deep learning architecture, CNN-GRU, combining a convolution neural network to capture the spatial variability of TEC, and a gated recurrent unit for temporal variability modelling ”
h) “From the tables given in the paper [14], one can see that different methods can give the best results for different stations” replace with:” As shown in table III in Kaselimi et al. [14], different methods can give the best results for different stations”
i) “….global scale was developed in [17]” replace with:” …global scale was developed in the work of Cesaroni et al. [17]
j) “The paper [18] gives an…” replace with: ”The work of Natras et al. [18] provides an… ”
k) “…was originally introduced in [4] and become…” replace with: ” …was originally introduced by Hochreiter et al. [4] and become…”
l) “…originally introduced in [5]” replace with: ” …originally introduced by Cho et al. [5]”
m) “…improves the results compared to [19].” replace with: ” …improves the results compared to the ones of Kharakhashyan et al. [19]. ”
n) “… by the example of [18] ” replace with: ” …by the example reported in the work of Natras et al. [18] ”
In line 47 the solar index F10.7 could be better specified writing for example that it is the daily value of the solar radio flux at 10.7 cm (F10.7) wavelength (Tapping 2013). K.F. Tapping, The 10.7 cm solar radio flux (F10.7). Space Weather 11, 394–406 (2013). https://doi.org/10.1002/swe.20064
- In line 105 “ME=-0.12 TECU”, the meaning of ME is not clear.
- From line 105 to line 112 “These values were….and 2.81 TECU, respectively” I suggest to rewrite better this part.
- From line 157 to line 167 sometimes you write VTEC sometimes you write TEC, please check.
- In line 319 is not clear what the postfix “RFF” and “FRF” indicate. RFF stands for?. FRF stands for?.
- In line 202 you should specify better the index Np because at http://omniweb.gsfc.nasa.gov/form/dx1.html you can download sigma_Np data, you should also define the acronym Vsw
- You should define the meaning of PReLU in line 331 instead of in line 344, besides it is not clear in figure 8 which parameter should be indicated in parenthesis PReLU (0,9) and PReLU (0,3).
- I suppose that R2 in line 384 and in Table 1 should be written as R2 , moreover MAPE values should be indicated with the symbol %
- In figure 11 it would be better to move the writing “architecture” below the x axis.
- From line 402 to line 404 you write: “The usage of bidirectional architecture essentially improves accuracy of the forecast and gives for all the architectures!!! and all stations values of MAPE less than 0.3 TECU, RMSE are less than 0.5 TECU, МАРЕ are less than 5 %.” I observe that MAPE is written two times, probably the first time is MAE not MAPE. Moreover, looking at figure 11, I do not understand why MAPE/MAE? should be less than 0.3 TECU, RMSE less than 0.5 TECU and МАРЕ less than 5%, I would say that these numbers hold only for the last bidirectional architecture, that is BiTCN
I strongly recommend to the authors to write a paragraph entitled “ Statistical Metrics Adopted in the Validation Process” where all the statistic parameters used to estimate the accuracy of TEC predictions are provided with their own formulas.
Comments for author File: 
Comments.pdf
I recommend to the authors a revision of their paper by a native English speaker.
Author Response
Responses to reviewer #1
We thank the reviewer#1 for his or her valuable comments.
- The meaning of the acronyms GRU (Gated Recurrent Unit) must be given in line 41 instead of line 113.
Response: Название аббревиатуры перенесено
- The meaning of the acronyms LSTM (long short-term memory), TCN (Temporal Convolutional Networks), CODE (Center for Orbit Determination in Europe), ANN (Artificial Neural Network), IGS (International GNSS Service), MAE (Mean Absolute Error), GIM (Global Ionosphere Map), SSN (Solar Sunspot Number), MAPE (Mean Absolute Percentage Error), RRN (Recurrent Neural Network), UPC, ED-LSTME, cGAN, JPL, must be written explicitly in the paper.
Response: The meaning of all acronyms was written explicitly in the paper.
Reply to comments about links to references
- a) Line 45: “An example of using the LSTM architecture for TEC prediction is the paper [8], which…” replace with: “An example of using the LSTM architecture for TEC prediction is the paper which….”
Remark taken into account.
- b) Line 50: “In [9], the authors developed and applied ….” replace with: “Sun et al. [9] developed and applied..”
Remark taken into account.
- c) Line 56: “In [10], the BiLSTM algorithm was applied” replace with:”Sivakrishna et al.. [10] applied the BiLSTM algorithm to forecast one hour ahead the Indian….”
Remark taken into account.
- d) Line 71: “In [11], several architectures ……are presented in Appendix of paper [11]” replace with: “ In Chen et al [11], several architectures…… are presented in the appendix….”
Remark taken into account.
- e) Line 85: “In [12], TEC forecasting model based on deep …” replace with: “ A TEC forecasting model based on deep learning was proposed by Tang et al. [12], it consists….”
Remark taken into account.
- f) Line 113: “Another conventional model – Gated Recurrent Unit (GRU) was compared in [13] with LSTM model results from equatorial station MAL2 data in Kenya and was more accurate than LSTM and other methods (Multilayer Perceptron (MLP), GIM_TEC and the IRI-Plas 2017) “ replace with: Iluore and Lu [13] analysing data from the equatorial station MAL2 (Kenya) found that GRU was more accurate than LSTM, Multilayer Perceptron (MLP), GIM_TEC and the IRI-Plas 2017.
Remark taken into account.
- g) Line 122: “In [14] a spatiotemporal deep learning architecture CNN-GRU combining a convolution neural network (CNN) to capture the spatial variability of TEC, and a gated recurrent unit (GRU) for temporal variability modeling has been proposed.” replace with:” Kaselimi et al. [14] proposed a spatiotemporal deep learning architecture, CNN-GRU, combining a convolution neural network to capture the spatial variability of TEC, and a gated recurrent unit for temporal variability modelling ”
Remark taken into account.
- h) Line 135: “From the tables given in the paper [14], one can see that different methods can give the best results for different stations” replace with:” As shown in table III in Kaselimi et al. [14], different methods can give the best results for different stations”
Remark taken into account.
- i) Line 144: “….global scale was developed in [17]” replace with:” …global scale was developed in the work of Cesaroni et al. [17]
Remark taken into account.
- j) Line 157: “The paper [18] gives an…” replace with: ”The work of Natras et al. [18] provides an… ”
Remark taken into account.
- k) Line 272: “…was originally introduced in [4] and become…” replace with: ” …was originally introduced by Hochreiter et al. [4] and become…”
Remark taken into account.
- l) Line 279: “…originally introduced in [5]” replace with: ” …originally introduced by Cho et al. [5]”
- m) Line 426: “…improves the results compared to [19].” replace with: ” …improves the results compared to the ones of Kharakhashyan et al. [19]. ”
Remark taken into account.
- n) Line 440: “… by the example of [18] ” replace with: ” …by the example reported in the work of Natras et al. [18] ”
Remark taken into account.
Reply to main comments
- In line 47 the solar index F10.7 could be better specified writing for example that it is the daily value of the solar radio flux at 10.7 cm (F10.7) wavelength (Tapping 2013). K.F. Tapping, The 10.7 cm solar radio flux (F10.7). Space Weather 11, 394–406 (2013). https://doi.org/10.1002/swe.20064
Response: the solar index F10.7 was specified.
- In line 105 “ME=-0.12 TECU”, the meaning of ME is not clear.
Response: The meaning of ME parameter (the mean error) was inserted.
- From line 105 to line 112 “These values were….and 2.81 TECU, respectively” I suggest to rewrite
better this part.
Response: The paragraph has been rewritten.
- From line 157 to line 167 sometimes you write VTEC sometimes you write TEC, please check.
Response: All designations of VTEC have been replaced with TEC.
- In line 319 is not clear what the postfix “RFF” and “FRF” indicate. RFF stands for?. FRF stands for?.
Response: Appropriate clarification was inserted.
- In line 202 you should specify better the index Np because at
http://omniweb.gsfc.nasa.gov/form/dx1.html you can download sigma_Np data, you should also
define the acronym Vsw
Response: Both of these parameters were specified.
- You should define the meaning of PReLU in line 331 instead of in line 344, besides it is not clear in
figure 8 which parameter should be indicated in parenthesis PReLU (0,9) and PReLU (0,3).
Response: Appropriate clarification was inserted.
- I suppose that R2 in line 384 and in Table 1 should be written as R2, moreover MAPE values should
be indicated with the symbol %
Response:Appropriate changes have been made. In addition, the R parameter was replaced by the correlation coefficient ρ.
- In figure 11 it would be better to move the writing “architecture” below the x axis.
Response: The writing “architecture” was moved below the x axis.
- From line 402 to line 404 you write: “The usage of bidirectional architecture essentially improves
accuracy of the forecast and gives for all the architectures!!! and all stations values of MAPE less
than 0.3 TECU, RMSE are less than 0.5 TECU, МАРЕ are less than 5 %.” I observe that MAPE is
written two times, probably the first time is MAE not MAPE. Moreover, looking at figure 11, I do
not understand why MAPE/MAE? should be less than 0.3 TECU, RMSE less than 0.5 TECU and МАРЕ
less than 5%, I would say that these numbers hold only for the last bidirectional architecture, that is
BiTCN.
Response: Detailed explanation inserted in abstract, after Figure 11, and in Conclusion.
- I strongly recommend to the authors to write a paragraph entitled “ Statistical Metrics Adopted in
the Validation Process” where all the statistic parameters used to estimate the accuracy of TEC
predictions are provided with their own formulas.
Response: A paragraph 2.2.6 entitled “Statistical Metrics Adopted in the Validation Process” including formulas was inserted.
Additionally, the following changes have been made:
(a) 5 papers have been added to the Introduction section to make the paper more relevant to the topic of Special Issue,
(b) the structure of the paper was brought into line with the journal template, while the methods used and description of the results obtained are presented in more detail, the Conclusion section is rewritten.
(c) Figures have been changed in accordance with the comment of the second reviewer.
With respect and gratitude
Authors: A. Kharakhashyan and O. Maltseva
Reviewer 2 Report
Please see the attached file.
Comments for author File: Comments.pdf
please see the attached file.
Author Response
Responses to reviewer #2
We thank the reviewer#2 for his or her valuable comments.
1.Line 35 ‘since its values determine the accuracy of positioning’.
In fact, there are many factors that affect the positioning accuracy, and the ionosphere delay is just one of them. The world ‘determine’ is not suitable, please modify it.
Response: Added among other factors.
- Line 36 and Line 37 ‘methods using neural networks stand out [3]. Their variety is enormous,’ These two sentences are not very coherent, please revise them.
Response: Sentences have been replaced:
Currently, among the methods of forecasting the ionospheric parameters, methods using neural networks are distinguished as one of the most diverse and undergoing active development [5].
- In the Introduction section, the author only described the content of TEC prediction in the previous papers and did not provide their own opinions. It is recommended to add a brief summary on the previous papers, such as which method is more suitable in which region, and what are the advantages of bidirectional, to better lead to the following text.
Response: Our opinion in the form of a short summary is inserted after line 179:
Thus, the analysis of literature data shows that there is a certain sequence of using neural network methods LSTM→GRU→ Bi-directional→TCN, which allows gradually increasing the accuracy of the TEC forecast. However, the results can be highly dependent on the combination of architectures, region, and space weather conditions.
In Sections 2, 4, an explanation of the advantages of bidirectional architectures has been added.
- Since the title mentions the European region, in the section 2 of this paper, it is necessary to describe the characteristics of TEC in the European region and how it differs from other regions.
Response: The behavior of the TEC is illustrated in Figure 2 and its additional description. The difference from other regions lies in the magnitude of the maximum and minimum TEC values, in the response to geomagnetic disturbances.
- Only four stations from the European region were selected in the article (Juliusruh, Murmansk, Moscow, Nicosia). Can these four stations represent an entire European region?
Response:The results are illustrated mainly by the example of the central European station Juliusruh, which is a reference station in terms of ionospheric studies and is still most often used in theoretical and experimental works. Due to sufficiently large latitudinal and longitudinal correlation radii of the TEC parameter, one can get an idea of the TEC behavior in a large zone. To obtain results outside the correlation radii, data from the high-latitude Murmansk station and the low-latitude Nicosia station are used.
- The TEC data used by the authors is JPL GIM-TEC maps, and then interpolated to obtain TEC data from four stations. The data from the stations was not used, so whether to use the station data to calculate TEC is better, after all, the map TEC is not the real value.
Response: You are right that we are not using real values, but values calculated from maps. This is because so far there are regions with very sparse coverage by GPS receivers (Africa, Russia), where maps are the only source of TEC data.
- The authors used three years, 2015, 2020, 2022 to conduct experiments. Would the chosen time span affect the final results? Please explain it.
Response: In accordance with this comment, the following changes have been made:
Line 219: inserted description of the differences in the behavior of TEC associated with the influence of solar activity in different years.
Line 414: a description of the dependence of the forecast accuracy on the year of observation is inserted.
- Line 387 Please provide a specific description of the meanings and formula expressions of the four metrics MAE, MAPE, RMSE, and R2, providing a reference for readers who are not familiar with deep learning.
Response: A paragraph 2.2.6 entitled “Statistical Metrics Adopted in the Validation Process” including formulas was inserted also in accordance with the comment of the first reviewer.
- From the results, it can be seen that using a bidirectional architecture can provide higher accuracy results. Please provide a more detailed analysis of the reason for this situation in section 4.
Response: In accordance with this comment, the following changes have been made:
Line 398: a description of the dependence of the statistical characteristics of the forecast on the combination of architectures is inserted.
In Sections 2, 4, an explanation of the advantages of bidirectional architectures has been added.
- The authors also compared their own results with the work of others in the conclusion section, but it was not comprehensive enough. If possible, it is recommended to have a more comprehensive discussion, such as regional differences, time span, methods, etc.
Response: More comprehensive discussion was added in the section 4. Discussion.
- There are some mistakes in words and figures that need to be corrected, like ‘meridian 30° N’ and the axis in Figure 1. The style of the figures needs to be unified.
Response: Line 17, abstract, line 102, line 219, caption to Figure 2: 30° N replaced with 30° E.
The style of the figures was unified.
With respect and gratitude
Authors: A. Kharakhashyan and O. Maltseva
