Probabilistic Evaluation of the Multicategory Seasonal Precipitation Re-Forecast

Round 1

Reviewer 1 Report

The manuscript titled “Probabilistic Evaluation of the Multicategory Seasonal Precipitation Re-forecast” discussed the predictive skills of the Meteo-France seasonal forecasting system 7, based on Brier score. The study is interesting and the results are practical significance for the prediction. Therefore I recommend that the paper be accepted after major revision.

1.       Why did the author analyze six methods? What differences?

2.       In the paper, the author used a new evaluation score, that is, BS. The present study only introduced the spatial distribution of the Brier score. I think that other results based on the existing metrics (such as Nash–Sutcliffe efficiency coefficient) should be given. And the advantages of this BS should be also displayed in the discussion.

3.       What were the errors over the tropical Pacific / Indian Ocean derived from?

Author Response

Please see the attachement.

Author Response File: Author Response.pdf

Reviewer 2 Report

The manuscript discussed the uncertainty issue of probabilistic evaluation of the multicategory seasonal forecast based on the precipitation re-forecast of Meteo-France seasonal forecasting system 7 (MF-SFS7) by Brier score. And the analysis method and conclusions are worth to refer with some scientific sense. The manuscript has some questions and concerns for minor revisions to clarify.

 1.      The manuscript should be improved in the introduction chapter (especially the last paragraph) to clarify the scientific question and novelty of this work in details.

 2.      It is recommended to add some comparison of the probabilistic evaluation results from MF-SFS7 with other seasonal forecast system (if could be provided from literatures such as ECMWF’s SEAS5) .

3. Some typos need to correct like relad in line 517.

Author Response

1.

I have clarified the scientific questions and novelty of this work in the introduction.

L144- 150:

….in the previous researches, this study aims to assess the robustness of the BS on seasonal time scale. Multiple analyses were performed to identify the system deficiency as well as to demonstrate the sensitivity of the BS to different analysis methods. The uncertainty of the BS derived from several analyses was quantified by the confidence interval, considering not only the classical binary precipitation events, but also the uncertain events. This article has 4 sections. Section 2 introduces the forecast system, the observation data, the methods of evaluation in this study. Section 3 describes and compares the results for seasonal and intra-seasonal analyses. Section 4 summarizes the main findings and conclusions obtained in this study.

2. 

The Johnson’s paper about ECMWF’s SEAS5 shows the precipitation ACC and CRPSS maps in JJA. A comparison of spatial distribution pattern of BS in TerA, TerB, and CRPSS, we found that both the BS in TerA and CRPSS have large errors and negative values in Nino 1 and 2 region, except the TerA map generated by Method 5. As the two scores and many other conditions are different, a direct comparison is not suitable. There are few published results can be compared directly.

L447-449:

The probabilistic errors, along with the highest ACC, in JJA seasonal precipitation hindcast are also identified over the Nino 1 to 2 region by Johnson et al. [20] with the ECMWF’s SEAS5, using the CRPSS evaluation score.

3. This has been corrected. I have checked the spelling and grammar in the manuscript.

Author Response File: Author Response.pdf

Reviewer 3 Report

Review of the article: "Probabilistic Evaluation of the Multicategory Seasonal Precipitation Re-forecast"

The paper is generally nicely laid out, and presented. The overall quality is good and meeting the minimum requirements of the journal. In my opinion, the contribution can become an important resource for the meteorology community.

I strongly suggest to author consider the following comments:

1) It would be important to mention what is the benefit of using GPCP precipitation data instead of using observational data from measurement stations. It is clear that the GPCP data is in grid points and that makes verification easier. However, it should be shown in some way that this data set represents reality efficiently.

2) It would also be important to say if there is any reason to use the GPCP (1993-2015) data instead of the GPCP Version 3.2 Satellite-Gauge that have higher resolution. It is very important that the author answer to this comment.

3) The figures (1, 2, and 5) are distorted and should be improved significantly.

4) The author does not demonstrate an adequate validation process. I recommend using a validation scheme of their research using ROC analysis or another approach like taylor diagrams, correlation map, etc.

Author Response

Please see the  attachement.

Author Response File: Author Response.pdf

Round 2

Reviewer 1 Report

Agree to accept.

Reviewer 3 Report

The manuscript has been revised. No further comments.

This manuscript is a resubmission of an earlier submission. The following is a list of the peer review reports and author responses from that submission.

Round 1

Reviewer 1 Report

The manuscript evaluates the seasonal precipitation re-forecast by Meteo-France seasonal forecasting system 7 with Brier score. The author used all the 7 figures in this paper to show the spatial distribution of the Brier score with different tercile thresholds for different samples, including 3-months precipitation anomalies, and 3-months mean precipitation of different ensembles. The figures are repetitive. The main conclusion is that the spatial distribution of the Brier score depends on the tercile thresholds and sampling methods, which have long been established (e.g., Bradley et al., 2008). This should be a good report for assessment of Meteo-France seasonal forecasting system 7, but unfortunately it currently does not achieve the standard of a scientific paper. For all these reasons I feel that the manuscript should be rejected in its actual form. I think the topic is potentially interesting, so I encourage the authors to investigate more in depth the performance of the forecasting system, analyze the simulations in more detail and create a new manuscript.

Reference:

Bradley, A. A., Schwartz, S. S., & Hashino, T. (2008). Sampling uncertainty and confidence intervals for the Brier score and Brier skill score. Weather and Forecasting, 23(5), 992-1006.

Reviewer 2 Report

I have a couple of concerns with the presented introduction/ methodology that should be addressed. There are several issues. I would like to discuss with the authors:

• The authors should explain the novelty of this work or improvement with respect to the previously derived methodology for the same study region. What is the uniqueness of the proposed methodology and its potential impacts, over other established state-of-the-art (e.g. machine and deep learning algorithm-based) techniques? What were the previously established data merging techniques available to forecast rainfall? The authors should explain with a couple of new paragraphs on this aspect in the introduction section. Also, you need to provide more literature reviews in terms of other rainfall forecasts associated with the research gap/limitation.

Yagmur et el. 2020. Modeling Level 2 Passive Microwave Precipitation Retrieval Error Over Complex Terrain Using a Nonparametric Statistical Technique. IEEE Transactions on Geoscience and Remote Sensing . DOI: 10.1109/TGRS.2020.3038343

Yin et el. 2021. Blending multi-satellite, atmospheric reanalysis and gauge precipitation products to facilitate hydrological modelling. Journal of Hydrology, 593, p.125878.

Zhang et el.. Merging multiple satellite-based precipitation products and gauge observations using a novel double machine learning approach. J. Hydrol. 2021, 594, 125969.

Ehsan et al. 2019 "Machine learning–based blending of satellite and reanalysis precipitation datasets: A multiregional tropical complex terrain evaluation." Journal of Hydrometeorology 20.11 (2019): 2147-2161.

• Extreme climate and weather events lead to a wide range of impacts on the society and environment and pose serious challenges. The authors should explain the impact of extreme climate on the study region with proper citations. Your overall introduction lacks clarity and does not lead correctly to a certain problem statement. 
• Your study area has a high dependency on local climate, and topographic complexity. Can you provide detailed climatic information for the selected study areas?
• You used multiple datasets with different sources with different temporal resolutions. How do you merge all those datasets into a common time series, please explain? It is necessary to report how the matching is carried out. This can have a significant impact on the results. I suggest adding a table that will include information about the resolution(temporal/spatial),
• Can you provide a high impactful schematic diagram to understand the proposed research framework where the big impact of the results can be presented?
• No seasonal analysis is performed. Can you show few results?
• Your error analysis results are incomplete. Can you provide a table for error analysis in terms of systematic error (absolute mean relative error) and random error (normalized root mean square error) for the evaluation results?
• The discussion must be included, and more references added. In the introduction, you mention articles that can be used to discuss and compare your results.