Reliability and Validity of a Chinese Version of the STEM Attitude Scale for Primary and Secondary School Students
Round 1
Reviewer 1 Report
Thank you very much for having the opportunity to review the paper entitled “Reliability and Validity of Chinese Version STEM Attitude Scale for Primary and Secondary School Students”. The authors tried to validate the Chinese version of the STEM Attitudes Scale for primary and secondary school students. Several recommendations and comments are provided.
Try to recheck on how citations are format for Chinese authors; for example: line 27 – “Ruan kunliang” might be the full name of the author. This should be (Ruan, 2003), same applies for the rest of the references, such as: Dong Chen & Qi Lan, 2013 – line 70, Fan Yaqin et al., - line 71, and so on.
Line 73 – this section, I assumed is for the research objective, better present the objectives by listing them for clarity. For instance, …. The objectives of the current study are as follows: …
Section 2.1 – sampling used? Inclusion criteria? IRB review? Since participants are minor, ethical consideration of the student should be stated clearly.
Line 101 – this paragraph should be revisit/rephrase for clarity. For instance, what is highest education? in line 107.
2.2 instruments or measures section - this should be rewritten and expanded, only limited information provided. Should provide why such demographics are included – gender, grade level, and parents’ educational attainment. Should also clarify the three factors – there be some sort of definition and description for each of the factors/variables used. There are many sample papers that the authors can looked into. For instance see: https://www.mdpi.com/2227-7102/11/9/481
Page 3 – see section about measurement instruments.
Statistical analysis should be improved. Many commonly used criteria and practice for EFA and CFA were not used. For instance, please go over: https://www.frontiersin.org/articles/10.3389/fpubh.2018.00149/full
This paper might be too strict and detailed. However, author/s can have a look on the various cutoff values and necessary test.
Alternatively, please check this paper:
Schreiber, J. B., Stage, F. K., King, J., Nora, A., & Barlow, E. A. (2006). Reporting structural equation modeling and confirmatory factor analysis results: A review. Journal of Educational Research, 99(6), 323-337. https://doi.org/10.3200/JOER.99.6.323-338
(This is one of the most highly cited paper on how to report CFA using SEM).
In addition, common practice now for CFA are at the least these should be reported, construct validity and reliability should be computed. Composite reliability, convergent validity (using the average variance extracted, AVE), discriminant validity (computed by taking the square root of AVE), and heterotrait-monotrait ratio of correlations (HTMT) of the variables/factors should be accomplish. Since, you are using AMOS you can incorporate Gaskin’s estimand for these computations (see: https://drive.google.com/drive/folders/0B3T1TGdHG9aEbFg1eEpqOWtrR3c?resourcekey=0-3O4AGIsjphE8yn037sRqHA
For the macros that can be incorporated into AMOS)
This you test for skewness and kurtosis? How about normality of the data? – This is important. Recommend remedy by using bootstrapping method, which is easily available in AMOS.
Correlation of the covariances should also be reported, if any (should include the changes of Chi-square values). Missing values? Should be clear on how were missing values treated.
See the following references for more information, this should be incorporated into the paper:
Ho, R. Handbook of univariate and multivariate data analysis and interpretation with IBM SPSS; Taylor and Francis: New York, 2006.
Fornell, C.; Larcker, D.F. Evaluating structural equation models with unobservable variables and measurement error. Journal of Marketing Research 1981, 18, 39-50, doi:10.2307/3151312.
Anderson, J.C.; Gerbing, D.W. Structural equation modeling in practice: A review and recommended two-step approach. Psychological Bulletin 1988, 103, 411-423, doi:10.1037/0033-2909.103.3.411.
Bollen, K.A. Structural equations with latent variables; Wiley-Interscience: New York, NY, 1989.
Hu, L.-T.; Bentler, P.M. Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal 1999, 6, 1-55, doi:10.1080/10705519909540118.
Furthermore, since the author is focusing on the validation of the instrument, further analysis should also be done. Such as the model should also consider presenting the “single factor” and the model with the “higher order factor”.
Line 138 – the use of critical ration – this should be cited
P should be “p” in italics. Chi-square notations should be check – line 167 not “X2” the 2 should be super-script, it is also not actually “x” but “χ”, you can use the word “Chi-square” alternatively.
Again the tables can be revisit and reformatted – see the above mentioned papers, there are many samples on how factor loadings are presented.
J8 loading of .449 - please check
As noted previously, Cronbach alpha is not enough for validation studies.
In general, the paper is interesting, however, the technical procedures for validation of a scale is not that strict and should be revisit. There are many exemplars available that the author could have a look. Criteria and common practice in presentation of results – specially the use of SEM/AMOS have evolved through the years.
Author Response
We really learned a lot from your careful and thoughtful comments and suggestions. You corrected our misunderstandings of presenting our research. Previously, in our mind, we needn't care too much about the goal presentation while conducting the validating paper like ours. And now we know clearly that we'd better stand at the readers' points to present our research more logically and clearly. The same with our contributions in this study comparing with the previous research work. We also know that the research articles of recent 3 years are very important. We tried our best to add the literature review of the recent 3 years to our paper as an independent part, however, due to very limited time, we are afraid that our review can not cover all relevant research work. We are also touched by your careful examining. Yes, Table 2 is really large, we took great effort to make it smaller. In conclusion, thank you very much for your enlarged our research vision and deepened our research depth with those constructive suggestions and thoughtful comments, with which we have better polished our research. We really appreciate your review.
Author Response File: 
Author Response.pdf
Reviewer 2 Report
Greetings,
In the introduction, emphasize the goals of the paper and the scientific contribution of the paper. Insert literature review selection. In this selection, look to use references in the last 3 years. In the results, Table 2 is too large. Try to correct it. See how others did it. Insert the data from Tables 4 and 6 into Table 3, in order to reduce the number of tables. In conclusion, give guidelines for future research and give the shortcomings of this study.
All best.
Author Response
Your valuable advice with great modesty touched us a lot. It is us who should say “thank you” for all of your meaningful comments and suggestions. You've made not only many detailed comments, at least 12 comments and suggestions, but also many constructive references which inlude both research articles and acdemic links.
We really learned a lot not just in the STEM research field. You suggest us to enhance the research foundation including the research objectives, sampling, criteria, ethical consideration, and so on, which improved our research greatly. You also helped us even polish the language with insightful questions. As we are not native English speaker, your patient tolerance and constructive comments always enlighened us a lot. So, we have made lots of improvements accordingly.
Please feel free to let us know if there are still any problems, we are always ready to have your thoughtful ideas. Thanks again!
Author Response File: Author Response.pdf
Round 2
Reviewer 1 Report
Thank you for inviting me to review the revised version of this paper.
In general the author/s already did their best to provide the necessary statistical information to make the paper more suitable to its goal of scale validation. Further clarifications are also provided with regards to the background of the scale and research procedure (including ethical statements and data collection).
Just a minor area - with regards to the discussion - the author/s should also include some discussions for the gender differences, grade level differences, and parent educational attainment differences that were found in the study. I would think that these are actually good finding and should be further included in the discussion.
In addition, some of the tables are quite confusing - author/s could try to look for simplier ways to present the analyses. Also advisable to have a first language speaker go over the paper.
Author Response
Thanks a lot for your approval and encouragement. We really learned a lot. We have new understanding that we’d better also discuss the new meaningful findings since we have found them although we mainly test the validity and reliability of STEM Interest Scale in Chinese context. Meanwhile, we’d better improve the table format as well as polish the content discussion.
- Just a minor area - with regards to the discussion - the author/s should also include some discussions for the gender differences, grade level differences, and parent educational attainment differences that were found in the study. I would think that these are actually good finding and should be further included in the discussion.
Response: Thank you very much for your meaningful suggestions. We have taken our effort to include discussion the impact of gender, grade and parent educational attainment differences on students STEM interests, which can be seen from Line 474 to Line 530 in the newly revised manuscript. It is really valuable to discuss more on the differences and reasons for it can guide the educators to take suitable measures to cope with the universal decline of STEM interests of primary and secondary school students.
- In addition, some of the tables are quite confusing - author/s could try to look for simplier ways to present the analyses. Also advisable to have a first language speaker go over the paper.
Response: Thank you very much for your advice. We separate Table 3 into two parts, the current Table 3 and Table 4, to make the results be more readable. And also, we checked all the tables to make them much easier to be read with shorter terms and good format. For example, we replaced “scientific interest” with “science interest”, and used “maths interest” instead of “mathematical interest” in all the tables and content.
Author Response File: Author Response.pdf
Reviewer 2 Report
Greetings,
The author complied with the comments of the reviewers. The paper is good now. However, the text on page 4 needs to be corrected to fit the existing text boundaries. In my opinion, Table 3, which was crossed out, was better than the new table. Results
ss - degrees of freedom
χ2 is a chi-square
SRMR - standardized root mean-square residual
RMSEA - root mean-square error of approximation
CFI - comparative goodness-of-fit index
NFI - normed goodness-of-fit index
AFS - goodness-of-fit index
AGFI - Adjusted goodness-of-fit index
put below the table or in a new table. Do not put these results in the results of factor analysis.
Limitations and further research suggestions fit into existing text boundaries.
All best.
Author Response
Thanks a lot for you beneficial suggestions. We understand much better through your great comments and advice that as researchers we’d better not only present our ideas and arguments orderly, but also design the format clearly. We have to reflect the new model and compare the new table with the old one so as to have a more suitable one. It is equally important to show the content and format in many cases.
1.The author complied with the comments of the reviewers. The paper is good now. However, the text on page 4 needs to be corrected to fit the existing text boundaries.
Response: Thank you very much for your recognition and encouragement. We corrected the text on page 4 to fit the existing text boundaries. Following your advice, we also corrected the similar problems in the manuscript.
- In my opinion, Table 3, which was crossed out, was better than the new table.
Response: We really appreciate your comparison and improvement advice. We also compared the new table with the previous Table 3. The new table is crowded with small numbers because more indicators are included in the newly added columns. So, we replaced the new table with the previous Table 3, and added a new table, Table 4, right after Table 3 since the previous comments suggested us to tell those indicators. After then, we gave the following table new number order after Table 4.
- Results
ss - degrees of freedom
χ2 is a chi-square
SRMR - standardized root mean-square residual
RMSEA - root mean-square error of approximation
CFI - comparative goodness-of-fit index
NFI - normed goodness-of-fit index
AFS - goodness-of-fit index
AGFI - Adjusted goodness-of-fit index
put below the table or in a new table. Do not put these results in the results of factor analysis.
Response: Thank you for you detailed result presentation suggestions. We borrowed your presentation format and reported x2/df, RMSEA, GFI,CFI, IFI, TLI, AGFI in a new independent table, namely Table 4, which really help us a lot in clearly presenting the results. The results are more readable now.
- Limitations and further research suggestions fit into existing text boundaries.
Response: Thank you very much for your valuable advice. We have made the limitations and further research suggestions fit into the existing text boundaries. We also corrected other similar problems in the manuscript.
Author Response File: Author Response.pdf
