New Design Method of Solid Propellant Grain Using Machine Learning
Round 1
Reviewer 1 Report
The submitted work deals with the application of a machine learning method to the optimal design of solid propellant grains. The general problem of design optimization for these grains, by accounting for the geometric parameters and their impact on the profile area (burn-back analysis) and the neutral/regressive/progressive character of this profile, is presented. The problematic is well described and justifies clearly the test of machine learning methods to ease and accelerate this optimization procedure. The considered machine learning method is a kernel-based SVM with gaussian kernel function: this classification method is first used to separate appropriate and inappropriate sets of configuration variables (according to the standard deviation on the profile area), which sounds logical, and then to optimize the configuration variables (considering the margin as a criterion), which is much more surprising and should be discussed more deeply.
From a general point of view, this work is interesting, the use of the method seems new and it could interest many researchers in the community. For that reason, this work is worth being published. Unfortunately, the submitted version still contains confusing sentence, redundancies and lacks some clarifications on the last step of the methodology. For that reason, major revisions are proposed and some of them are detailled below.
- all along paper: authors are invited to submit the paper to a native English speaker. Some sentences are confusing, the paper also still contains mistakes (missing s, extra s...). Some of them are described below, but a complete verification is necessary.
- line 14: "requirements is": no S
- line 14: "various grain configuration": missing S
- line 21: "reducing the difficulty the optimum": missing word ?
- 33-35: "However..." confused sentence
- 39 "has a high difficulty" ?
- 64-65 "... a model that learns the correlations... ": no, SVM used here is for classification, not for regression. This point rises other questions that create confusion all along paper. Classical SVM is used for classification, but is is right that it can be used for regression, but is then the margin the good criterion ? It is not clear if the margin is the criterion or a constraint to separate good and bad sets of variables. Authors should definitely clarify this point for the article to be useful to any reader. Authors indicate that the margin indicated the difference from the objective: so, a margin of 0 should be good ? But the margin is then maximised... It would very very relevant to develop a bit the presentation of the SVM method and give the proper definition of the margin, since this feature is a central feature of the method.
- 74-84: this section should not be in the introduction, it sounds like a conclusion. Either it is not at the right place, or it should be rewritten.
- 89-90: redundant sentence.
- 98-108: this section should be in the introduction.
- 122: N ".. star branch": branches.
- 176: "requirements": please be accurate with the terms. Do these requirements concern the optimization criterion or the constraints ?
- 190: concerning the database: have the burn-back analyses been performed for all of these 49500 data ? If so, it could be interesting to cite the best ones: that could be interesting for comparison with the final optimal configuration ?
- 202: why a gaussian Kernel ?
- 203-209: this part should be developed, it is the basis of the originality of the paper, but the description is too short for interested readers.
- 214: among the 49500 sample data, how many are infeasible ? line 222, 11524 samples are classified "1" and 7564 are "-1": what about the others ?
- 233: C is usually an hyperparameter of SVM: how does the method learn it ? The role of gamma is not clear.
- 234: these values yield an error rate of 0 %. It sounds good but is there no risk of overfitting ? What would be the consequences for new designs generated close to the hyperplane, or in uncertain zones ?
- 237: Figur 4, not 5. Please indicate values of w, f and epsilon in the caption.
- 248: 4 test cases are presented. What is the overal quality on other cases ?
- section 4.3: this section is difficult, especially concerning the use of the margin as an optimization criterion. Its role as a constraint is clear, but as a criterion, it is different: authors say the margin is constant away from the hyperplane, so that creates flat (non-sensitive) criteria.
- 279-280: redundant sentence.
- table 6: is there any alignment issue between the columns ? In the constraints column, w is fixed to 44, but its value then becomes 95.18 or 46.18...
- 287: "the initial value is the end of the variable range" ?
- 290-291: "the simplex method with SVM searches for the maximum margin": authors should demonstrate that tha mximum margin is also the minimum standard deviation. A parity plot built with a the database samples should be informative. At this point, the reader is not convinced by the relation between margin and performance.
Author Response
The authors are deeply grateful to the reviewer for thoughtful comments. The authors faithfully revised the manuscript according to the reviewer’s opinions, thereby improving the quality of the manuscript.
- all along paper: authors are invited to submit the paper to a native English speaker. Some sentences are confusing, the paper also still contains mistakes (missing s, extra s...). Some of them are described below, but a complete verification is necessary.
- line 14: "requirements is": no S
- line 14: "various grain configuration": missing S
- line 21: "reducing the difficulty the optimum": missing word ?
- 33-35: "However..." confused sentence
- 39 "has a high difficulty" ?
A) The entire manuscript has been corrected in English, including the grammar pointed out. Finally, English proofreading was performed through a professional proofreading institution.
- 64-65 "... a model that learns the correlations... ": no, SVM used here is for classification, not for regression. This point rises other questions that create confusion all along paper. Classical SVM is used for classification, but is is right that it can be used for regression, but is then the margin the good criterion ? It is not clear if the margin is the criterion or a constraint to separate good and bad sets of variables. Authors should definitely clarify this point for the article to be useful to any reader. Authors indicate that the margin indicated the difference from the objective: so, a margin of 0 should be good ? But the margin is then maximised... It would very very relevant to develop a bit the presentation of the SVM method and give the proper definition of the margin, since this feature is a central feature of the method.
A) SVM was used to classify classes. The standard deviation is a value of how close the burning surface area profile is to the neutral. The stored data are classified into classes based on the standard deviation 0.5. If the standard deviation is less than 0.5, it is a neutral, and if it is greater than 0.5, it is a non-neutral burning surface area profile. SVM learns the hyperplane, which is the boundary of the class, from the stored data. Margin is the difference between Classification target and the hyperplane. “Margin = 0” is the hyperplane, and hyperplane is the criterion for classifying the class. Therefore, “margin=0” replaces “standard deviation = 0.5”, and is the criterion for class 1 and class -1. Margin is used to classify classes. It is not used for the purpose of obtaining the smallest standard deviation. The margin of the Gaussian kernel SVM is different from that of the linear SVM. In this study, the characteristics of the margin were used to eliminate local solution and modify the grain configuration. These details have been supplemented in the manuscript so that the revised content can be easily understood.
- 74-84: this section should not be in the introduction, it sounds like a conclusion. Either it is not at the right place, or it should be rewritten.
A) The manuscript was revised to be appropriate for the introduction to reflect the opinions of the reviewers.
- 89-90: redundant sentence.
A) It has been deleted from the revised manuscript.
- 98-108: this section should be in the introduction.
A) The part was modified according to the reviewer’s opinions.
- 122: N ".. star branch": branches.
A) It is corrected.
- 176: "requirements": please be accurate with the terms. Do these requirements concern the optimization criterion or the constraints ?
A) In this study, there are two requirements for grain design. The first is to classify the configuration variables with neutral burning area profile. The neutral burning area profile is defined when the standard deviation is less than 0.5 in this study. The second is to change for any configuration variables to new one that has a neutral profile that is most similar to the original configuration. The burn-back analysis for all design sets must be performed to determine the first requirement, but this study succeeded in replacing the burn-back analysis with the machine learning from the database. The authors apologized that the description of the second requirement is insufficient and has been corrected to be understood more clearly. The starting point for grain design may have been previously designed or may be set in other disciplines. In this study, the objective is to modify a configuration with a similar burning surface area profile, but with a smaller standard deviation. To achieve the objective, the set of configuration variables must be modified in the direction of increasing the margin. Therefore, in this study, the simplex method was used to search for the margin gradient around the starting point, and the set of configuration variables were modified until class 1 was clear (margin ≥ 0.0106). These are supplemented in the revised manuscript.
- 190: concerning the database: have the burn-back analyses been performed for all of these 49500 data ? If so, it could be interesting to cite the best ones: that could be interesting for comparison with the final optimal configuration ?
A) The 49500 data was obtained through a burn-back analysis for all designed conditions in advance. The obtained data is classified by the burning surface area profile with a neutral (class 1) and a non-neutral (class -1). The learning technique performed in this study is not to derive the final configuration directly, but to predict whether a new set of configuration variables is neutral or not. This has been confirmed to be of great help in deriving the optimized configuration.
- 202: why a gaussian Kernel ?
A) Gaussian kernel is one of the representative kernels of SVM. It is suitable for learning nonlinear boundaries because it can reliably learn complicated boundaries. As a result of analyzing the boundary of the class database obtained through the burn-back analysis, it was confirmed that the boundary is difficult to simulate with Linear, Quadratic and Cubic, and the authors determined that a Gaussian kernel would be suitable. As a result of performing SVM using Linear, Quadratic and Cubic kernels and Gaussian kernels, only Gaussian kernels learned hyperplane properly. These are supplemented in the revised manuscript.
- 203-209: this part should be developed, it is the basis of the originality of the paper, but the description is too short for interested readers.
A) In SVM using a Gaussian kernel, C and γ are important variables that determine the shape of the hyperplane. C determines the range of support vectors used in the hyperplane calculation. If C is large, the range narrows and the number of support vectors decreases. γ determines the effect of the support vector on the hyperplane. When γ is small, the hyperplane is greatly affected by the support vector, and the shape of the hyperplane changes rapidly. Using small C and γ produces a complex and fine hyperplane, but overfitting is prone to occur. Large C and γ ignore fine boundary shapes and learn coarse hyperplanes. Therefore, the error can be reduced only by using the appropriate C and γ for the data. The effect of γ and C on the hyperplane is described in detail in Ref. [15]. In this study, the authors used Bayesian optimization provided by MATLAB to obtain C and γ with minimum error rates. These contents supplemented with the revised manuscript.
- 214: among the 49500 sample data, how many are infeasible ? line 222, 11524 samples are classified "1" and 7564 are "-1": what about the others ?
A) Of the 49500 data, 11524 are class 1 and the remaining 37976 are class -1. The reason for the decrease in the number of Class -1 used for training is to exclude data not used for training. Applying all the data to training increases training time, but does not affect classification performance. Therefore, in this study, 30412 unnecessary data were excluded by data mining. The manuscript has been supplemented so that the content can be clearly understood.
- 233: C is usually an hyperparameter of SVM: how does the method learn it ? The role of gamma is not clear.
A) The role of γ answered above in detail. MATLAB used in this study provides a function to optimize the hyperparameter so that the error rate is lowered by applying Bayesian optimization, and the authors utilized it.
- 234: these values yield an error rate of 0 %. It sounds good but is there no risk of overfitting ? What would be the consequences for new designs generated close to the hyperplane, or in uncertain zones ?
A) In general, the main cause of overfitting occurs when the boundaries between classes are unclear or when inappropriate C and γ are used. In this study, the authors tried to apply machine learning to the solid propellant grain design. In the propellant grain design, the standard deviation of the burning surface area is varied with the configuration variables, but no noise or singularity is generated. Therefore, the saved data is clustered by class, so it can be clearly classified. In this study, C and γ of the Gaussian kernel obtained the lowest error rate through the Bayesian optimization. Various margin contours were analyzed to determine if overfitting occurred, but no overfitting was found. The database of this study was construct through a combination of geometric variables at regular intervals. Therefore, if one support vector is deleted, the deleted area is not supplemented. Because of this condition, the error rate increases when overfitting prevention techniques such as Holdout validation are applied. Therefore, in this study, learning and error rate calculations were performed using all data mining results.
For data that are very close to the hyperplane, there is a possibility of mispredicting the class. In order to reduce the uncertainty region, it is necessary to be small the interval of the configuration variable, but it can be inefficient because it increases the time to construct and learn the database. Therefore, in this study, margins were used to design the class as a clearly 1 region. In Section 4.3, “Margin = 0.0106” is the smallest margin of the support vector. The support vector is clearly neutral because it is selected from data with class 1 in the database.
- 237: Figur 4, not 5. Please indicate values of w, f and epsilon in the caption.
A) The typo was corrected, and the caption was revised.
- 248: 4 test cases are presented. What is the overal quality on other cases ?
A) This is a similar comment as mentioned above. Except for very few cases that are very close to the hyperplane, all sets of configuration variable have been classified correctly.
- section 4.3: this section is difficult, especially concerning the use of the margin as an optimization criterion. Its role as a constraint is clear, but as a criterion, it is different: authors say the margin is constant away from the hyperplane, so that creates flat (non-sensitive) criteria.
A) There was a lack of explanation for the design process using a margin. In section 3.3, the characteristics of margin are described. When using the margin calculated by the Gaussian kernel, the local solutions is deleted. Therefore, the difficulty of the optimization problem can be reduced. Margin increases from the support vector of class -1 to the support vector of class 1. Therefore, optimization techniques were used to increase the margin of the set of configuration variables. The criteria for ending the search were mentioned above.
- 279-280: redundant sentence.
A) Sentences with the same meaning have been removed.
- table 6: is there any alignment issue between the columns ? In the constraints column, w is fixed to 44, but its value then becomes 95.18 or 46.18...
A) The table was corrected properly.
- 287: "the initial value is the end of the variable range" ?
A) The initial value is the starting condition for the simplex method, and the case of optimal design failure using standard deviation was used. When the starting point is the center, a neutral configuration was designed, so a new starting point was set as the end of the variable range, and a starting point converging in the local solution was obtained. In this study, the inner radius and fillet radius used the upper boundary of the design range, and the angular coefficient used the lower boundary of the design range.
- 290-291: "the simplex method with SVM searches for the maximum margin": authors should demonstrate that tha mximum margin is also the minimum standard deviation. A parity plot built with a the database samples should be informative. At this point, the reader is not convinced by the relation between margin and performance.
A) The margin and the standard deviation are different. "Margin = 0" and "standard deviation = 0.5" are almost the same, but margins and standard deviations can be different in other region. In Section 3.3, the information that shows the relationship between the two has been added.
Reviewer 2 Report
In this article, the authors propose a design method using machine learning. It is suitable for the grain configuration design in a practical system. The model is trained using the support vector machine.
Before I start my review of the text, I would like to point out that the level of English language in the text is low.
The way the text is approached does not meet the specifications of a journal with high scientific requirements. There are many grammar, syntactical and punctuation errors. In several parts of the text, it does not make sense.
The authors provide insufficient information to the reader.
The methodology followed is not documented and, this raises many questions.
The paper is not well-organized. It does not have the proper structure, readability and length.
The technical contribution is low and not clear.
Specifically, I have to make the following remarks:
1)The abstract does not have the proper structure and has many unnecessary details. It should have been organized as follows: 1)background, 2)motivation, 3) gap challenges, 4) proposed approach, 5) evaluation and results from 6) significance.
2)The "Related Work" section is required after the introduction. Ηow your study is different from others? What they have where others do not? Why they are better or how? My suggestion is to increase the number of up-to-date references to achieve deeper discussion and comparison with the state-of-the-art.
3)The authors claim to apply ML techniques. This venture was not successful. What are the criteria for choosing SVM? References [14-17] are not enough. A scientific article needs documentation (algorithmic, mathematical and state-of-the-art comparison).
A background section about ML algorithms and techniques is necessary.
4)What data set was used to implement the model? What are the quality features? In addition, what is the implementation environment? The authors simply state that they used the Matlab tool.
5)The results seem unlikely to attract much interest from readers. The way they are presented does not provide any substantial innovation.
6)In the discussion, the authors didn't mention the limitations and the potential issues of this study.
7)Ιn the conclusion section, It is not stated any numeric information related to accuracy and other advantages of the proposed solution.
Moreover, conclusions could discuss future research directions and extensions of this study.
8)Finally, the authors haven't included up-to-date references and from MDPI as well.
The article has many weaknesses and can not be published
Author Response
The authors are deeply grateful to the reviewer for thoughtful comments. The authors faithfully revised the manuscript according to the reviewer’s opinions, thereby improving the quality of the manuscript.
1)The abstract does not have the proper structure and has many unnecessary details. It should have been organized as follows: 1)background, 2)motivation, 3) gap challenges, 4) proposed approach, 5) evaluation and results from 6) significance.
A1) The abstract was modified according to the reviewer’s opinion.
2)The "Related Work" section is required after the introduction. Ηow your study is different from others? What they have where others do not? Why they are better or how? My suggestion is to increase the number of up-to-date references to achieve deeper discussion and comparison with the state-of-the-art.
A2) The main topic of grain design research is to develop the optimization techniques. This is a nonlinear problem that is complex and difficult to analyze, and the studies have been conducted to find easier and faster solution using an appropriate optimization technique than to correlate relationship between the configuration variables and burning surface area profile. Therefore, studies are being conducted to find a general optimization technique suitable for grain design [1~3, 5~8], and currently, the method of using multiple optimization techniques at the same time [8] is being discussed as the most reliable and accurate method. Design using optimization techniques is performed through iterative method, so it is important to design and analyze new shapes quickly in previous studies. As a result, all results in the intermediate process are discarded, and it is difficult to design a complicated configuration that takes a lot of analysis time. In this study, the authors tried a new approach to adopt machine learning the correlation between the configuration variables and the burning surface area profile rather than the developing appropriate optimization techniques. In the process of developing a real solid propulsion system, a large amount of data is generated through iterative calculations until a final solution is obtained. Such data could not be utilized if the design objective were different in the previous design process, but the use of stored data is possible with machine learning, making design easier and faster. In the solid propellant grain design, no studies have been performed to apply machine learning of the method that proposed in this study. The descriptions have been supplemented in the introduction.
3)The authors claim to apply ML techniques. This venture was not successful. What are the criteria for choosing SVM? References [14-17] are not enough. A scientific article needs documentation (algorithmic, mathematical and state-of-the-art comparison).
A background section about ML algorithms and techniques is necessary.
A3) The objective of machine learning is to classify whether a configuration has the neutral burning surface area profile or not. Databases have classes and set of configuration variables. This is a binary classification problem with 5 variables. The configuration variables are nonlinear problems with complex effects. In this study, it was determined that the gaussian kernel SVM, which can classify nonlinear classes as a type of supervised learning, is suitable.
The difference from traditional grain design studies is that the design is performed using margin. Margin is the criterion for classifying classes, but in this study, it was also used as a method to reduce the difficulty of design by eliminating regional solutions. Section 3.3 has been added to the manuscript to supplement these contents.
4)What data set was used to implement the model? What are the quality features? In addition, what is the implementation environment? The authors simply state that they used the Matlab tool.
A4) The data consisted of external radius, internal radius, web thickness, fillet radius, angle coefficient and class. To train the model classifying the class, 19088 data (class 1: 11524 + class -1: 7564) were used. The quality feature in this problem is a class. The class is classified whether it is less than or greater than the standard deviation 0.5. The size of the standard deviation is not considered. Margin is the criterion for classifying whether the class is 1 or -1. If the margin is negative, class is classified as -1. If 0 or positive, class is classified as 1. The equipment to train the model was a conventional office PC (windows 10, CPU 3.60 GHz, RAM 16 GB). The function of MATLAB was used for the machine learning and optimization.
5)The results seem unlikely to attract much interest from readers. The way they are presented does not provide any substantial innovation.
A5) The main topic in the field of solid propellant design is to develop general optimal design technology, which is going on in a way that need a lot of time and cost, such as the use of many high-performance optimization techniques. The primary reason is that it is difficult to study the correlation between the configuration variables and the burning surface area profile. However, while studying optimization techniques, a large amount of data has been generated and various software that can apply machine learning are being used. This is a suitable environment for studying correlations. The authors confirmed that the grain design is performed quickly and accurately with less calculations when the method proposed in this study is applied, and this is described in the manuscript. In author’s opinion, research institutes and companies that perform grain design of solid propellants already have a lot of data, and the method proposed in this study can be effective in reducing design costs by utilizing the stored data, so it is thought that it is worth utilizing enough. In this regard, the authors ask the reviewer to re-evaluate the academic value of this study.
6)In the discussion, the authors didn't mention the limitations and the potential issues of this study.
A6) This study is a fundamental study using machine learning and has studied in only limited region. The margin is a different concept from the standard deviation. Due to the nature of the Gaussian kernel, the minimum standard deviation cannot be designed. To minimize the standard deviation, an additional optimum design must be performed. In practical grain design, the requirements such as average burning surface area and burning time should be satisfied. In this study, a model was trained to classify whether the burning surface area profile is neutral. It has not been validated on how much the amount of sample data is valid. Learning with less data may also work. This limitation is a new topic of study. These contents have been supplemented with the revised manuscript.
7)Ιn the conclusion section, It is not stated any numeric information related to accuracy and other advantages of the proposed solution.
Moreover, conclusions could discuss future research directions and extensions of this study.
A7) The main achievement of this study is to confirm that machine learning is a useful technique in the field of grain design. Machine learning can learn useful information from stored data, and it can be used to design the desired grain configuration without additional burn-back analysis. In this study, classes were classified with high accuracy by applying the SVM technique through machine learning. Using the suggested method, it is possible to eliminate the local solutions and reduce the design difficulty, so this method can be useful. However, this study is fundamental, and only the neutral burning surface area profile of the single grain configuration is considered, so further research is required to obtain general grain design technology. There are a variety of study topics using machine learning, such as analyzing the appropriate number of data, multi class classification taking into account the average burning surface area and burn time, the training effect of a randomly generated database, and the application of ANN to calculate set of configuration variables that satisfy the minimum standard deviation. These contents were used to supplement the conclusion.
8)Finally, the authors haven't included up-to-date references and from MDPI as well.
A8) SVM is a suitable method for fault diagnosis. Among the latest papers of MDPI, two papers related to fault diagnosis using SVM have been added to the references.
Round 2
Reviewer 1 Report
Authors have significantly improved the quality of the paper by adding very pedagogic contents concerning the method used. A new example has been introduced to help the reader understand the reasons why authors chose this criterion for optimization. Authors answered all comments proposed by the reviewers and added useful comments in the paper. The work is worth being published.
Reviewer 2 Report
