Reinforcement Learning for the Face Support Pressure of Tunnel Boring Machines
Round 1
Reviewer 1 Report
This research investigates the application of the Deep Q-Network reinforcement learning algorithm to control the tunnel face support pressure during excavation. The algorithm is tested against both analytical and numerical environments, with the analytical environment used for hyperparameter tuning. In order to investigate the algorithm capability to select the best sequence of face support pressures, the approach used the algorithm from DeepMind research group to adapt and test on random geologies.
The novelty of this method lies in the reinforcement learning approach, where the machine has no prior knowledge of the environment and is educated through rewards defined by the developer, with the simulation of environment using the finite difference method (FDM).
The results show that the model is able to optimize the face support pressure, given a sufficient number of episodes are present. The paper is written in a concise style with technical language, and the outcome is presented convincingly with thorough discussions. Although no real case study is presented in this research.
Author Response
Thank you very much for reviewing our manuscript. Since no specific issue was raised, there is no need for a point-by-point reply.
Reviewer 2 Report
The reviewer formulates the following recommendations for improvement of the article, which the authors must be executed for successful publishing (acceptance):
- the abstract is too short and too general, please supplement it by using methods common in academia and science;
- the introduction is quite short and poorly written. There are citations with grouping of 15-20 items. It is totally inaccurate. If someone checks, there are approx. 50 rows and 95 references in the introduction (particularly there are 50 references mentioned in 10 rows). Please prepare an adequate introduction with appropriate and accurate literature review. This Section – in this form – is not worthy of a high level of research;
- a nomenclature must be added after the List of abbreviations in which the author must give all the parameters, symbols, their meanings, and the applied accurate units;
- the List of abbreviations does not contain all the mentioned abbreviations in the text, please double-check it. E.g., NATM, AI, etc.;
related all the equations, i.e., after the equations, the authors must supplement the applied units in the explanation of the parameters, and symbols;
- please try to divide and separate much better the tables, e.g., Table 1, the content of the table cannot be understood well without further structuring;
- Table 1: what does "game over" mean? It would be a scientific publication...
- please explain it in a more detailed manner, why you have applied 50-50 neurons in the first and the second hidden layers?
- please use "–" instead of "-" for negative sign in the entire manuscript;
- the explanation of "u" parameter is missing from Equation (3);
- please explain, why you have considered exactly the following five possible support pressures (50, 100, 150, 200 and 250 kPa)?
- please explain the meaning of the parameter "a" in Equation (4), it seems to be missing;
- Figure 4: what does "–100" mean on the vertical axis? If nothing, please reformat the figure (subfigures); (see Figure 7, too);
- in Figure 4b: there are not 100 lines but the title of the figure states it, please explain;
- Figures 5b, 5d and 5f: the charts cannot be seen well, please reformat the figures; it is the same situation with Figure 8;
- title of Figure 5: it does not contain the explanations/meanings of all the subfigures, please supplement them; it is the same situation with Figure 8;
- title of Figure 6: what does "e" subfigure mean? Is it the "c"?
- Figure 10: what are the units of the length values (dimensions), please supplement them;
- Figure 11: a legend is missing from the figure, please settlement;
- Figure 14: what is the difference between the settlement (Figures 5 and 8, and the parameter "u") and the vertical displacement values? why did you apply different signs for them? Please explain and try to eliminate the possible confusing situation;
- Section 4 (Discussion): what does "C51" mean? Please explain in a more detailed manner;
- the link "https://github.com/soranz84/220721_TBM_RL" does not work: neither with a point at the end of the link, nor without is (see Data Availability Statement);
- Would it be a possibility to calibrate and validate your "model" with field tests and connecting measurements?
- What do you think about dynamic modeling? Or considering dynamic effects?
Author Response
We would like to express our gratitude for your thoughtful and constructive review of the manuscript. Your insights and feedback have been invaluable in improving the quality and clarity of our work.
We appreciate the time and effort you invested in providing detailed comments and suggestions, and we found your critiques to be both insightful and helpful. Your expertise and attention to detail have undoubtedly contributed to making our paper stronger.
The point-by-point replies to your comments are given in the attached pdf.
Author Response File: 
Author Response.pdf
Reviewer 3 Report
The manuscript presents the use of the reinforcement learning algorithm to select the best sequence of face support pressures in the case of boring machines in tunnel excavation. The authors consider the problem when the machine has no knowledge of the environment it is exploring. The paper presents a description of the method, examples, discussion of results and conclusions.
In the opinion of the reviewer, the paper may be published in the Geosciences, but the authors should make corrections and explanations presented below:
1. line 2 down from 79 (there are no line numbers between 79 and 80 on page 2) - no explanation of Si+1
2. table 2 - could the authors explain how the variation was calculated.
3. Figures 4, 7 and 12 - maybe in the figure or in the caption the authors should describe what the orange band means, because the description has to be found in the text.
4. Page 9 – Is it possible to show the presented algorithm in the form of flowchart? it would be more understandable to the audience.
5. Page 10 - could the authors explain how they performed the sensitivity analysis? Were the calculations performed for only three values? If so, are the authors sure that it is enough?
6. Figure 8 - the caption is not correct. The figures are denoted from a to f, and only points a, b, and c are in the caption.
7. Line 444 - the given data page doesn’t work.
8. references 126 to 139 are not cited.
In my opinion, the paper can be considered for publication in the Geosciences after minor revision.
Author Response
We would like to express our gratitude for your thoughtful and constructive review of the manuscript. Your insights and feedback have been invaluable in improving the quality and clarity of our work.
We appreciate the time and effort you invested in providing detailed comments and suggestions, and we found your critiques to be both insightful and helpful. Your expertise and attention to detail have undoubtedly contributed to making our paper stronger.
The point-by-point replies to your comments are given in the attached pdf.
Author Response File: Author Response.pdf
Round 2
Reviewer 2 Report
With the fulfilled improvements, the reviewer recommends the publishing of the paper after fine/minor English language proofreading.
Congratulations on the work!
Author Response
Your dedication and commitment to the task are truly appreciated, and we are grateful for the time and effort you have invested in this process. It is because of your contribution that we were able to meet our deadline successfully.
Once again, thank you for your excellent work, and we look forward to working with you again in the future.
After proofreading, we have made minor text edits according the table below.
Author Response File: Author Response.pdf
