Modified Neural Architecture Search (NAS) Using the Chromosome Non-Disjunction

Round 1

Reviewer 1 Report

1. The authors use an extremely low crossover rate. An explanation for such a value is needed. 2. Is the complexity of the proposed approach to improve results by 0.7% justified? 3. Are the obtained results Average 0.74582 0.75026 0.75292 0.75158 statistically significant? 4. Authors should carefully review the article for technical errors, for example line 231, page 7 - "Algorithm 1" should be "Algorithm 2".

Author Response

Answer to Reviewer1

 

Modified Neural Architecture Search (NAS) using the Chromosome Non-disjunction

 

 

By Kang-moon Park, Donghoon Shin, Sung-do Chi

Manuscript ID: applsci-1351446

 

We acknowledge reviewer’s helpful comments and suggestions. They are very important for the improvement of the quality of our manuscript. All the comments and suggestion are incorporated into the revised manuscript or explained why not.

 

1. The authors use an extremely low crossover rate. An explanation for such a value is needed.;

 

(Answer) Parameters of Table 2 are based on our previous paper. However, they are tuned because complexity of experiment is improved. We have performed several times of experiment in order to determine those parameters. In terms of cross-over rate, experiments using values greater than 0.05 showed poor convergence.  As you mentioned, we added some sentences in Section 4.B:

 

These parameters are tuned based on our previous research [41] because of improvement of complexity of problem domain. We have performed several times of experiment in order to optimize those parameters.

 

2. Is the complexity of the proposed approach to improve results by 0.7% justified?;

 

(Answer) According to your advice, we explain the meaning of improvement by 0.7% in the last paragraph of Section 5.B and first paragraph of conclusion. :

 

Although it seems to be low improvement compared to the effort, however, it can be substantiated that it is such a meaningful results since it has been achieved without human experts.

 

It can be said that tuning of the already exist DNN without the human experts is the main contribution of our research.

 

3. Are the obtained results Average 0.74582 0.75026 0.75292 0.75158 statistically significant?;

 

(Answer) Those average and maximum values are the results of 50 times of independent runnings. We added explanation of it.:

 

Figure 9 demonstrates the model accuracies of 50 times of independent running of the model.

 

4. Authors should carefully review the article for technical errors, for example line 231, page 7 - "Algorithm 1" should be "Algorithm 2".;

 

(Answer) As you mentioned, we did some mistakes and we reviewed again all the way through, by revising those errors.

Author Response File: Author Response.doc

Reviewer 2 Report

The topic of the paper is interesting. It can be considered for publication, however, there are issues to be addressed.

• Line 91: use either abbreviation or full name. please be consistent
• Line 102 needs reference
• The caption for figure 1 is not clear. It should be complete so one can easily understand the entire flowchart without the need to go into the text
• your first case study is Multilayer perceptron ANN with BP training, right? the explanation for that is not very clearly written.
• For case study I, I suggest you compare the performance of your proposed method with another method as a baseline to compare.
• For case study II, you have reported the accuracy. I highly recommend that you also report the computation time. It is important that we can see if the improvement in accuracy is justified considering the computational complexity. 
• The architecture of the model and its complexity is very like to the complexity of the data. Please refer to Section 6 of this review paper (https://doi.org/10.1007/s00170-021-07325-7) that talks about the challenges associated with deep learning. Hyperparameter and architecture tuning and the achieved performance may be affected by the size and nature of the data. In fact, the complexity of data and the definition of an acceptable performance could be field-dependent (in some engineering applications, for example, 90% accuracy is perfect while in many cases, 99% is considered as good). So, these factor shows that whether it is worth to make the fine-tuning automated, depending on the improvement in performance (affected by model size and complexity) and the computational cost. Please, address this point in your discussion. 
• FOr future research, I suggest, studying the impact of the dataset (the nature, size, complexity) on the performance of your proposed model. also, you may consider other heuristic optimization methods. It is interesting that you want to employ the model on RNN, but I suggest that you compare the performance with some conventional methods to compare the performance of your proposed model. 

I wish the authors the best of success in their research and would be pleased to review your revised manuscript. 

 

Author Response

Answer to Reviewer2

 

Modified Neural Architecture Search (NAS) using the Chromosome Non-disjunction

 

 

By Kang-moon Park, Donghoon Shin, Sung-do Chi

Manuscript ID: applsci-1351446

 

We acknowledge reviewer’s helpful comments and suggestions. They are very important for the improvement of the quality of our manuscript. All the comments and suggestion are incorporated into the revised manuscript or explained why not.

 

1. Line 91: use either abbreviation or full name. please be consistent.;

 

(Answer) As you mentioned, we reflect your comment by using abbreviation for the consistency and coherency of the manuscript:

 

2. Line 102 needs reference;

 

(Answer) As you mentioned, we added reference of that statement. :

 

3. The caption for figure 1 is not clear. It should be complete so one can easily understand the entire flowchart without the need to go into the text:

 

(Answer) In order to make it clear, caption of figure 1 has been modified:

 

>> Fig. 1. System configuration of NAS using genetic algorithm with chromosome non-disjunction.

 

4. Your first case study is Multilayer perceptron ANN with BP training, right? the explanation for that is not very clearly written.:

 

(Answer) According to your comment, we added explanation of first experiment in the first paragraph of section 4.A. :

 

>> Due to the fact that the used dataset has low complexity, a simple Multiple Layer Perceptron (MLP) with error backpropagation has been implemented in this experiment.

 

5. For case study I, I suggest you compare the performance of your proposed method with another method as a baseline to compare.:

 

(Answer) Purpose of the case study 1 is not supposed to compare performance, but to verify converging performance which shows identical results to our previous study. In order to explain purpose of the case study, we added some sentences in first paragraph of Section 4.A. :

 

>> This case study is intended to further analyze our previous study[62]. The purpose of the case study 1 is to verify whether starting with a different network converges to the same result as the previous study.

 

6. For case study II, you have reported the accuracy. I highly recommend that you also report the computation time. It is important that we can see if the improvement in accuracy is justified considering the computational complexity.:

 

 (Answer) As you mentioned, we added the computations time (sec) in table 3. :

 

Accuracy	Original Model	Model (a)	Model (b)	Model (c)
Average	0.74582	0.75026	0.75292	0.75158
Max	0.7536	0.7534	0.7582	0.7566
Computation time(sec)	1.2.	1.2	1.2	1.21

 

 

7. The architecture of the model and its complexity is very like to the complexity of the data. Please refer to Section 6 of this review paper (https://doi.org/10.1007/s00170-021-07325-7) that talks about the challenges associated with deep learning. Hyperparameter and architecture tuning and the achieved performance may be affected by the size and nature of the data. In fact, the complexity of data and the definition of an acceptable performance could be field-dependent (in some engineering applications, for example, 90% accuracy is perfect while in many cases, 99% is considered as good). So, these factor shows that whether it is worth to make the fine-tuning automated, depending on the improvement in performance (affected by model size and complexity) and the computational cost. Please, address this point in your discussion.:

 

(Answer) Thanks for your comments, we read through the review paper you recommended and it helped a lot to improveour conclusions. As you mentioned, we added some sentences for data sensitive problem of NAS is added in the  third paragraph of Section 6:

 

>> However, it has a limitation in that it consumes too many costs such as time and resources compared to improved accuracy. Since it has data sensitivity issues which is directly affected by the size of the dataset available [65]. Auto-tuning of the deep neural network is such a still time consuming job when it comes to deal with massive amount of the real dataset.

 

8. For future research, I suggest, studying the impact of the dataset (the nature, size, complexity) on the performance of your proposed model. also, you may consider other heuristic optimization methods. It is interesting that you want to employ the model on RNN, but I suggest that you compare the performance with some conventional methods to compare the performance of your proposed model.:

 

(Answer) According to your suggestions, we are excited to consider further experiments and approaches. We think your suggestion will be very helpful for improving our future research. :

 

>> Studying the impact of the dataset by considering the nature, size, and complexity on the performance of the proposed approach is the topic of our future research. In order to more clearly substantiate the improved performance of our proposed model, compare with some conventional methods is another topic of our future research.

 

Author Response File: Author Response.doc

Round 2

Reviewer 2 Report

The given comments are addressed in the revised manuscript and I recommend the paper for publication.