Testing Commercial Intrusion Detection Systems for Industrial Control Systems in a Substation Hardware in the Loop Testlab

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

This paper focuses on the integration of Industrial Control Systems (ICS) with Information and Communication Technology (ICT) systems, highlighting the increased cyber-attack risks due to this integration and the need for specialized security measures. It presents a novel experimental protocol for evaluating Intrusion Detection Systems (IDS) in a digital substation environment, revealing significant variations in IDS responses to industrial-specific cyberattacks and underscoring the challenges in establishing consistent performance baselines in dynamic laboratory settings.

- Line 20: "As Power Grid play" should be "As Power Grids play" 

- Line 34: "Many analysis claims" should be "Many analyses claim"

- Line 75-78: The sentence "ICS Intrusion Detection Systems (IDS) is designed to detect cyber-attacks in ICS" should use "are" instead of "is"

- Given the large number of references to previous works, it's advisable for the authors to include a summary table at the end of the section. This table should list all the cited works along with their key characteristics. Where possible, it should also highlight the similarities or differences with your approach.

- Emphasizing the future work more prominently is essential to demonstrate the significance and potential impact of this research. It will also provide a clear direction for other researchers, guiding them in building upon and extending the findings of this study.

Comments on the Quality of English Language

I suggest the authors to thoroughly reread the entire paper and check the English for accuracy and clarity.

Author Response

Thank you for your thoughtful feedback and constructive comments. They have been very helpful in guiding our revisions and improving our manuscript.

-Line 20: "As Power Grid play" should be "As Power Grids play"

-Line 34: "Many analysis claims" should be "Many analyses claim"

-Line 75-78: The sentence "ICS Intrusion Detection Systems (IDS) is designed to detect cyber-attacks in ICS" should use "are" instead of "is"

We have updated the text based on your suggestions.

 

-Given the large number of references to previous works, it's advisable for the authors to include a summary table at the end of the section. This table should list all the cited works along with their key characteristics. Where possible, it should also highlight the similarities or differences with your approach.

Following your suggestion, we have added a summary table at the end of the related work chapter. This table includes a list of all cited works along with some of their key characteristics. This should better highlight similarities and differences with our approach, providing a clearer comparison and context.

 

-Emphasizing the future work more prominently is essential to demonstrate the significance and potential impact of this research. It will also provide a clear direction for other researchers, guiding them in building upon and extending the findings of this study.

We have incorporated a dedicated paragraph in our paper to specifically address future work. This addition aims to provide a clear direction for further research in this area. We hope this enhancement will assist other researchers in building upon and extending the findings of our study.

 

-I suggest the authors to thoroughly reread the entire paper and check the English for accuracy and clarity.

We have thoroughly reviewed and revised the paper to enhance its accuracy and clarity. In response to the reviewer's feedback, we have made moderate edits to improve the English language. Although the revisions may not result in perfect English, we believe there is a noticeable improvement in the paper's quality. We have highlighted all the changes made for easy identification.

Sincerly,
Jon-Martin Storm
University of Oslo

Reviewer 2 Report

Comments and Suggestions for Authors

-In the abstract, it does not present the main results.

-It does not reference tables 1 and 2 in the document.

-In figure 1, the information contained is not appreciated; it is suggested to make it larger.

-Table 2 should have the description of the information at the top of the table.

-At the end of table 3, the information "noindent1 Tables may have a footer" is not clear.

-Table 5 is not in the correct format.

-The description of figure 1 is too long; the title can be shortened, and the information discussed in the body of the document.

-Table 2 and figure 2 are located in the methodology section, but the content corresponds to results. It is suggested to move the table and figure, along with all the information related to results, to the appropriate section.

-The document contains many abbreviations, and some of them are not used again after being introduced.

-There are many abbreviations that do not appear in the list; the following are just a few examples: DMZ line 333, SIEM line 456, ASDU line 300, TP line 406, FP line 407, TN line 408, FN line 409, CR line 412, DR line 414, SIEM line 424, ENIP line 448.

-The document does not explicitly state which is the best commercial Intrusion Detection System.

Author Response

Thank you for your thoughtful feedback and constructive comments. They have been very helpful in guiding our revisions and improving our manuscript.

-In the abstract, it does not present the main results.

We have rewritten the abstract and  incorporated the main results of our study to provide a clearer and more concise summary of our research findings.

-It does not reference tables 1 and 2 in the document.

Table 1 and table 2 are now referenced in the paper.

-In figure 1, the information contained is not appreciated; it is suggested to make it larger.

Figure 1 has been changed to a whole page width figure to make the information easier to read.

-Table 2 should have the description of the information at the top of the table.

The caption for table 2 has been moved to the top of the table.

-At the end of table 3, the information "noindent1 Tables may have a footer" is not clear.

The footer was quite a glaring and embarrassing inclusion from our side. We have removed the footer.

-Table 5 is not in the correct format.

We converted the table into a figure to more effectively showcase the confusion matrix.

-The description of figure 1 is too long; the title can be shortened, and the information discussed in the body of the document.

We have updated the text based on the your suggestion.

-Table 2 and figure 2 are located in the methodology section, but the content corresponds to results. It is suggested to move the table and figure, along with all the information related to results, to the appropriate section.

We do not consider the attacks a part of the results, since they are selected from a set of attacks from another experiment [Erdodi et. Al. 2022]. However, we clearly see that this could be misunderstood in our presentation. The table and text has been updated to better show that it is a part of the method and not the result.

-The document contains many abbreviations, and some of them are not used again after being introduced.

In response to your feedback, we have carefully reviewed our document and made revisions regarding the use of abbreviations. We have also removed any abbreviations that were not used recurrently to enhance clarity and readability. Thank you for pointing out these areas for improvement.

-There are many abbreviations that do not appear in the list; the following are just a few examples: DMZ line 333, SIEM line 456, ASDU line 300, TP line 406, FP line 407, TN line 408, FN line 409, CR line 412, DR line 414, SIEM line 424, ENIP line 448.

In response to your feedback, we have carefully reviewed our document and made revisions regarding the use of abbreviations. 

-The document does not explicitly state which is the best commercial Intrusion Detection System.

In response to your query, we have clarified in our study that the rule-based ICS IDS, Omicron Stationguard, exhibited the best performance in our test lab. However, due to the dynamic and changing conditions of the test environment, we must treat this result with some caution. These conditions may have influenced the performance metrics, rendering a definitive conclusion about the best commercial IDS somewhat uncertain. Further research in a more consistent and stable environment is necessary to validate these findings.

Sincerly,
Jon-Martin Storm
University of Oslo

Round 2

Reviewer 1 Report

Comments and Suggestions for Authors

The requests have been executed by the authors, and in my opinion, this has improved the overall quality and clarity of the paper.