Image Segmentation for Mitral Regurgitation with Convolutional Neural Network Based on UNet, Resnet, Vnet, FractalNet and SegNet: A Preliminary Study

Round 1

Reviewer 1 Report (Previous Reviewer 3)

thanks for your answers

Author Response

Point 1:  thanks for your answers

Response 1: thanks for your sign to our reports

Author Response File: Author Response.pdf

Reviewer 2 Report (New Reviewer)

A new medical dataset is presented in this paper that can be used to predict cases of mitral regurgitation. Additionally, they use well-known segmentation neural network architectures to predict the disease and compare their results with the dataset. The paper is interesting, but it requires extensive English editing before it can be published. A strong related work section and a discussion section are also required. Some figures are not clear. Also, there is unnecessary capitalization, extra commas, and dots throughout the manuscript.

Additional comments:

Line #32. A comma appears twice after 93.46%.

Line #42. CARDIA, ARIC, and CHS. When mentioning it for the first time, please provide the full form, not an abbreviation.

Line #42. Cite the sources of the data.

Line #54. Automated Assessment of Mitral Regurgitation Using Color Doppler Echocardiographic Images... Does it need to be capitalized?

Line #72 - 76. The sentences are incomplete.

Line #79 - 82. The sentence structure is confusing and incomplete. Before publication, the manuscript needs extensive English revision.

Line # 85-135. There should be a related work section in the manuscript that includes these details. In this section, there is information that is unrelated to the paper, and the writing should be clear and concise. As I mentioned above, the paper requires extensive English revisions.

Compare these architectures with yours and explain why your design is necessary. By doing so, you will be able to strengthen your proposal.

Table 1. There is no mention of the number of videos in Normal 17.

For tables and figures, please provide a detailed caption. Readers should be able to understand what the authors are attempting to convey (a takeaway message) through the figure.

Line #200. Cite the Label Me annotation tool. Since the annotators are not medical practitioners, how reliable is the ground truth?

Line #238. I am unable to comprehend the sentence. What is ad? What is laplapin?

Line 250. Please explain the benefits of compression and path decompression.

Eq 3. The F1 score should be added in brackets to the Dice coefficient.

Eq 4. What is Pixel Accuracy nobjects? What is nobjects?

Figure 4. The figure is unclear. The text is difficult to read.

Table 5. The data is presented in multiple rows for many cells. Improve the readability of the table.

Line 382-390. It would be helpful if you included related works in one section and compared them with your design. If you intend to compare the results, be as concise as possible.

Table 8. Unreadable and distorted.

Discussion. You are repeating results that have already been reported. Alternatively, consider cases where all predictions were correct or predictions that were accurate for only a few models, along with your remarks as to why this occurred. Include some examples where all models failed to predict correctly, along with your reasoning.

Author Response

Please see the attachment

Author Response File: Author Response.pdf

Round 2

Reviewer 2 Report (New Reviewer)

My comments have been incorporated into the paper.  

This manuscript is a resubmission of an earlier submission. The following is a list of the peer review reports and author responses from that submission.

Round 1

Reviewer 1 Report

Summary

This paper makes an empirical comparison of different deep neural network architectures for segmentation of 2D color Doppler echocardiograms. The particular architectures considered were U-Net, SegNet, ResNet, Vnet U-Net2 and U-Net3. The goal was to identify heart mitral regurgitation defects on the input images. The authors collected a small sample for the test. The evaluation was based on pixel-level metrics for segmentation. Multiple architectures achieve very similar performance on this very small dataset.

Technical Details

-        - One thing that is not clear here is the kind of input images that these networks receive. The authors mention using Color Doppler and the example in Figure 2 already shows some that the input contains some highlighting around the heart issue which these networks are expected to identify. It is unclear what is the target application for this work if the input is an image which already highlights the patterns that need to be identified. It seems like a rather trivial problem.

-         - Figure 3 is being taken straight from [13] without even acknowledging the source. The quality of this image is even low with bad quality on the legend. The figure has been published by the IEEE and as such it is copyrighted. It should not be reproduced here without acquiring the permission to reuse it here and without proper acknowledgement of the source, otherwise it becomes a form of plagiarism.

-         -  Discussion Section. Authors make invalid direct comparisons against other methods such as [4], [22], [23] and [14]. These works used different datasets and/or evaluation metrics. No conclusions should be drawn from these.

-         -  Data size. The dataset is quite small, containing samples from just 20 patients. Using a fixed training and testing datasets, with 17 for training and only 3 for testing is not appropriated. It will be better to apply some sort of cross-validation instead to better measure the robustness of the proposed method.

-           -  Training. Authors show results for “testing data” as the network trains. One should not peak into the testing dataset during training, that is why validation sets are used instead.

 -         -  Images per video. Authors do not explain why there are many images for some patients and only a few for other patients.

-          -  From the results shown in Table 5, it can be seen that different networks are capable of achieve very similar performance here. In particular, the differences between Segnet and U-Net3 are almost negligible. Considering that the test set is very small, it is likely that these differences between metrics are not statistically significant. One cannot say that such metrics are enough evidence to “prove” that U-Net3 outperform other architectures. On the other hand, other related works have already provided better evidence for this.

-        - Data collection. The dataset includes examples of patients with mitral valve regurgitation and normal patients. However, these “normal” patients are those who are already “suspected of having heart abnormalities”. I believe that it is better to also include examples of people who do not have any suspicion of heart abnormalities, who are expected to be completely healthy.

-         - Organization. In the introduction, there is a brief discussion about U-Net being the preferred network architecture for multiple medical imaging applications. However, the paper comes back to a more general idea of “CNN being applied in various medical image segmentation tasks”. It is better to start first with the more general ideas that introduce the notion of using CNN in the medical field, and the move to the specifics of some architectures being used to solve specific problems.

-         - Organization. The order in which some ideas are discussed in Section 2.3 is less than ideal. Each network is presented as a separated work, but these have many things in common. For example, the general encoder-decoder is used by these networks under different names such as “shrinkage and expansion”, or “compression and decompression”. The authors should unify these and they should rather talk about what is common and what is different between these architectures.

-         - Section 2.4 about metrics appears in the middle of Section 2.3 about architectures. It is like a general picture about architectures is provided first, then the metrics are presented, and then the paper goes back to list the details on each architecture.

-        - What do the authors mean by “Segnet has the appropriate encoder and decoder network”?

- Sections 2.3.2 and 2.3.3 on ResNet and V-Net. These ideas were already mentioned in the first paragraph of Section 2.3. It is basically a repetition with some slightly different wording.

- Table 3. In the text, it is described that U-Net3 uses 3x3 convolutional filters. However, in this table the kernel sizes are huge and asymmetrical (64x63x1; 128x129x3). The whole description of the architecture here seems incorrect.

-     - Figure 4.  These plots do not have the same scale, and it becomes tricky to compare them visually. Some go from 0.0 to 1.0, other from 0.5 to 1.0, other from 0.9 to 1.0. It would be better if the minimum and maximum values are fixed for all networks.

LLanguage / Figures / Formatting

-         - There are some issues with the right use of punctuation. These need to be fixed.

-        - The long version is “et alia” and the abbreviation is “et al.”. Here, the dot in “al.” is typically not included.

-        - Table 1. This table is a bit of a waste of space. The same data could be presented in a way that saves more space. Also, the table right now is shown across two pages, this should always be avoided.

-        - Table 5. There are numbers highlighted which are not the highest values on their corresponding column.

-        - Author of reference 13 is “Daniele Liciotti”, not “Daniel” as spelled by the authors here. In any case, the work should be referenced using the last name “Liciotti et al.”.

-        -  The are also a few grammar errors. For example:

o   “the incidence rate of heart valve disease is the most common is mitral regurgitation”

o   “In the proposed method, the structure of the method”

o   “at the end of the laplapinThe encoder network on the Segnet”

o   “"which is slightly smaller the difference lies in the object segmentation. , the architectural model and the epoch used."

-       -   Fix issues in the bibliography with references: 5 (no year and publication venue), 8 (preprint is cited, but the paper has been published in TPAMI), 10 (preprint is cited, but the paper has been already published), 13 (this is an ICPR paper, not written by the “IAPR”), 18 (preprint is cited, paper published in 3DV).

Reviewer 2 Report

Atika et al evaluate the performance of different CNN structures including U-Net, SegNet, V-Net, and ResNet architectures for identifying and segmentation of mitral regurgitation. The paper has flaws in presentation including repetition (e.g., check tables 2 and 4). The methods are not clear and the robustness of the results can not be assessed. 

Reviewer 3 Report

1-Improve the quality of figures

2-Make sure your figures do not need any reference

3- Please check line 314

4- What is your innovation in this paper?

5-Improve the literature review

6-There are some grammatical errors. Please correct them

7- Your dataset is public or private?