Comparative Evaluation of Sustainable Framework in STEM Intensive Programs for Secondary and Tertiary Education
Round 1
Reviewer 1 Report
I enjoyed reading this manuscript on the development and analysis of the sustainable STEM education framework. As the authors state, this topic is trending globally and, therefore, it is important to analyze the framework to understand its effectiveness so that it could be shared and implemented across numerous international jurisdictions. I hope that the following feedback will assist the authors in improving a very important piece of literature that will benefit the STEM education field:
First, it is essential to define or operationalize "sustainable education development" in this paper. While the definition of STEM education is briefly clarified, "sustainable STEM education" and "secondary and tertiary STEM intensive programs" in this paper are not.
The method section seems out of sequence. While the PDCA model is introduced at the beginning, it isn't elaborated on until the very last few paragraphs. I found myself having to go back-and-forth between these two sections for it to make any sense.
The introduction of MOTIV and TechHub (lines 117-118 & 131-132) are confusing as written. I think it would be better to briefly name the two programs in the previous paragraph. The detailed descriptions of these programs give a really comprehensive overview.
What program are you referring to when describing the infrastructure provided by UPB (starting at line 153)? I have the same question for the learning objectives (starting at line 165) and pedagogical approaches (starting at line 183).
At line 191, I'm wondering what the 20 components are (I see this listed in a table on p. 9).
It is interesting yet refreshing to read the limitations of the programs at the beginning of the results section. It is important to set up the context of how these STEM programs operated so I appreciate that.
The discussion has merit. That being said, I feel as though there is still a broad piece missing from it. What do the results say about the need/value for sustainable STEM education frameworks? This is briefly mentioned (lines 445-447) but the big questions of "How?" and "Why?" are not answered. I would also like to see the framework spelled out or explained separately from its analysis. Based on the research conducted, what is the authors' final sustainable STEM education framework proposal?
Author Response
Point 1: First, it is essential to define or operationalize "sustainable education development" in this paper. While the definition of STEM education is briefly clarified, "sustainable STEM education" and "secondary and tertiary STEM intensive programs" in this paper are not.
Response 1: The concepts have been defined in the introduction chapter of the paper, as follows:
- sustainable education development – lines 40-45
- sustainable STEM education – lines 1128 - 132
- secondary and tertiary STEM intensive programs – lines 132 - 134
Point 2: The method section seems out of sequence. While the PDCA model is introduced at the beginning, it isn't elaborated on until the very last few paragraphs. I found myself having to go back-and-forth between these two sections for it to make any sense.
Response 2: According to the reviewers’ suggestion, this section has been restructured and now has a logical flow.
Point 3: The introduction of MOTIV and TechHub (lines 117-118 & 131-132) are confusing as written. I think it would be better to briefly name the two programs in the previous paragraph. The detailed descriptions of these programs give a really comprehensive overview.
Response 3: The two programs have been named in the previous paragraph (line 187) and their introductions have been restructured and clarified.
Point 4: What program are you referring to when describing the infrastructure provided by UPB (starting at line 153)? I have the same question for the learning objectives (starting at line 165) and pedagogical approaches (starting at line 183).
Response 4: The infrastructure is the same for both proposed programs, as it needs to be unitary throughout the framework. This is stated at lines 237-238.
The learning objectives are defined using Bloom’s revised taxonomy for both programs. This was omitted in the initial manuscript and is now inserted at line 246.
Pedagogical approaches as defined in the research are applied for both intensive programs. This was omitted in the initial manuscript and is now inserted at line 275.
Point 5: At line 191, I'm wondering what the 20 components are (I see this listed in a table
on p. 9).
Response 5: Table 2 has been referenced within the text of the paragraph - line 279
Point 6: It is interesting yet refreshing to read the limitations of the programs at the beginning of the results section. It is important to set up the context of how these STEM programs operated so I appreciate that. The discussion has merit. That being said, I feel as though there is still a broad piece missing from it. What do the results say about the need/value for sustainable STEM education frameworks? This is briefly mentioned (lines 445-447) but the big questions of "How?" and "Why?" are not answered. I would also like to see the framework spelled out or explained separately from its analysis. Based on the research conducted, what is the authors' final sustainable STEM education framework proposal?
Response 6: As suggested by the reviewer, the framework was described in the last section of the paper and additional details regarding the added value and implications were given in lines 601 – 632.
Author Response File: 
Author Response.pdf
Reviewer 2 Report
The literature review needs attention to its overall organisation. There are many aspects discussed (often rather briefly) and the links between the different aspects are not explored or explained fully. I suggest that the content here should be reviewed and presented in a more coherent thread. (e.g. P2 paragraph 1- it is not clear why work experience is being discussed- the links between this and early leaving of education need to be drawn out).
P2 line 81- define what is meant by a blended approach- is this a multidisciplinary approach or a blend of teaching techniques (e.g. practical, online etc)
Try to avoid describing studies conducted by others- what is important here is their findings and how these drive your argument forward.
The literature review is trying to cover too much content. It would be better to focus on the main and significant elements and explore these in more depth.
Later in the paper Bloom’s taxonomy is used a basis of some of the work. This should therefore be explored and/or reviewed in this section.
It would be useful to include some further information about the participants and how they were selected. Was the make up of the student groups the same for each project? Did the students do both projects? Etc.
It would also be useful to describe the content of the work in more detail. Perhaps some further details about one aspect of the work could be given to illustrate the content and aims of the work for the reader. Defining the gamification and complex reward system would be useful. Setting out the learning objectives would help with this. How was it decided what the most appropriate instructional strategies were?
The reader needs to have a much greater understanding of the interventions/projects used in order to be able to fully access and understand the results and implications of the work.
Why was a qualitative approach discounted? Would this not have offered more nuanced and detailed feedback? When discussing factors such as efficiency and value qualitative data may have offered valuable insights into the students experiences which could be triangulated with the overall statistics. In fact, in the discussion the paper does draw on some qualitative information from a discussion forum. It would have been useful to have been presented with a fuller analysis of this in the results section.
More detail about the data gathering survey is required. For example, how was the design of the 20 components in the quantitative survey decided upon? Was the approach based on previously designed and reviewed tools? Was the survey piloted? What questions were asked and why? How was the survey distributed, to who (and how many participated) when did the survey take place in relation to the running of the modules?
More detail of the analysis described in paragraph 2 (line 213) required, and again for the correlation matrix described in paragraph 3 is required. Why were these methods selected? How were the structural steps decided upon and how do these relate to the theory?
It is not entirely clear why learning style has been included in the matrix. Much of the theory here is disputed and again, as this is part of the research the theory should be explored in the literature review.
There is an assumption made that c-10 scored poorly due to the students not having a previous relationship with the professors. This assumption needs to be explored in more detail and evidenced. Would hiring high school professors incur a training cost that would also need to be included in the analysis?
It would be useful to summarise the main conclusions from the results and to discuss these alongside the limitations of the work.
Author Response
Point 1: The literature review needs attention to its overall organisation. There are many aspects discussed (often rather briefly) and the links between the different aspects are not explored or explained fully. I suggest that the content here should be reviewed and presented in a more coherent thread. (e.g. P2 paragraph 1- it is not clear why work experience is being discussed- the links between this and early leaving of education need to be drawn out).
Response 1: As suggested, the Introduction section has been restructured and appropriate reasoning has been added between all presented aspects.
Point 2: P2 line 81- define what is meant by a blended approach- is this a multidisciplinary approach or a blend of teaching techniques (e.g. practical, online etc)
Response 2: As suggested, the definition was inserted in the Introduction and further detailed in section 3.1 - Structural development of intensive programs – line 134.
Point 3: Try to avoid describing studies conducted by others- what is important here is their findings and how these drive your argument forward.
Response 3: As suggested, implications of each referenced study has been addressed in the Introduction section of the manuscript.
Point 4: The literature review is trying to cover too much content. It would be better to focus on the main and significant elements and explore these in more depth.
Response 4: As suggested, the introduction section has been expanded and now includes a more focused discussion based on the current literature review, corelated with authors own arguments.
Point 5: Later in the paper Bloom’s taxonomy is used a basis of some of the work. This should therefore be explored and/or reviewed in this section.
Response 5: As suggested, Bloom’s revised taxonomy was discussed in the Introduction section (line 106) and further detailed in section 2 - Materials and Methods (lines 246-252).
Point 6: It would also be useful to describe the content of the work in more detail. Perhaps some further details about one aspect of the work could be given to illustrate the content and aims of the work for the reader. Defining the gamification and complex reward system would be useful. Setting out the learning objectives would help with this. How was it decided what the most appropriate instructional strategies were?
Response 6: The gamification strategy for both secondary and tertiary intensive programs is an under-review book chapter submitted to the project partners for final evaluation. It is expected to be ready for publishing in summer of 2021. Nevertheless, the authors provided within this manuscript a summary of the used gamification and reward system for both programs (lines 221-233). Learning objectives are defined in Appendix A. Selection of the instructional strategy was done via a pilot course and the procedure is detailed starting with line 264.
Point 7: The reader needs to have a much greater understanding of the interventions/projects used in order to be able to fully access and understand the results and implications of the work.
Response 7: The full description of the projects and developed tools is offered for the reader in references 6, 7, 19, 20, and 35.
Point 8: Why was a qualitative approach discounted? Would this not have offered more nuanced and detailed feedback? When discussing factors such as efficiency and value qualitative data may have offered valuable insights into the students experiences which could be triangulated with the overall statistics. In fact, in the discussion the paper does draw on some qualitative information from a discussion forum. It would have been useful to have been presented with a fuller analysis of this in the results section.
Response 8: The full qualitative analysis is considered a limitation of the study, because of the process of translating the qualitative components into a quantitative evaluation using AHP in order to accurately correlate value costs. Nevertheless, the authors considered that this particular approach allows better reproducibility of the research results, consequently a very similar implementation of the proposed framework. This limitation is mentioned in the 4th section of the study, Discussion and Conclusions (line 622).
Point 9: More detail about the data gathering survey is required. For example, how was the design of the 20 components in the quantitative survey decided upon? Was the approach based on previously designed and reviewed tools? Was the survey piloted? What questions were asked and why? How was the survey distributed, to who (and how many participated) when did the survey take place in relation to the running of the modules?
Response 9: A description of the procedure for setting the components was inserted at line 400. The components were defined by the authors based on the learning objectives, thus the divided structure in overall and content specific components. Also, components considered the content of each curriculum module. A detailed description of the survey application was inserted in chapter 3.2. Efficiency evaluation - line 403.
Point 10: More detail of the analysis described in paragraph 2 (line 213) required, and again for the correlation matrix described in paragraph 3 is required. Why were these methods selected? How were the structural steps decided upon and how do these relate to the theory?
Response 10: The analysis was based on Bloom’s Taxonomy, and more detail was given in section 3, line 312 and lines 347 – 359. Based on the previous research conducted by the authors (references 6, 7, 19, 29 and 35) and their professional experience, curriculum design considered the correlation of content module with the following: learning objective, pedagogic approach/ instructional strategy, learning style, delivering format, assessment. This is also emphasized by research conducted in references (3, 17, 36, 37). The matrix was constructed using brainstorming and nominal group technique, corelated with resource (human, material, financial) requirements and availability of UPB. These details were inserted within the manuscript at lines 375 – 384.
Point 11: It is not entirely clear why learning style has been included in the matrix. Much of the theory here is disputed and again, as this is part of the research the theory should be explored in the literature review.
Response 11: Three basic types of curriculum design were identified by Roblyer M.D., namely: subject-centered, learner-centered, and problem-centered design. Higher education curriculum development is typically based on subject-centred approaches, which focuses on a particular subject or discipline, the model being less concerned with individual learning styles compared to other forms of curriculum design. One major drawback of this approach is that students who are not responsive to this model tend to fall behind, because of decreased levels of engagement and motivation. The proposed STEM framework focuses on blended learning, including learner-centered, and problem-centered design of curriculum content, thus, learning styles are of particular importance in these hypotheses. Furthermore, the design process of the framework appealed to eight learning styles based on psychologist Gardner’s theory of multiple intelligences, namely: visual; verbal; logical; auditory; social; intrapersonal; physical and naturalistic. For maximum learning efficiency, these learning styles were paired with learning objectives, delivery formats, instructional strategies/ pedagogical approaches, assessment methods and module content, as presented in Table 1, line 391.
The justification was inserted both in the introduction (line 148) of the article, as literature reference and in chapter 3.1 (line 367).
Point 12: There is an assumption made that c-10 scored poorly due to the students not having a previous relationship with the professors. This assumption needs to be explored in more detail and evidenced. Would hiring high school professors incur a training cost that would also need to be included in the analysis?
Response 12: This assumption was made based on the feedback given in the live sessions with high school pupils. They felt that the learning environment and approaches were very different to what they experienced in high school, to the extent of perceiving the teacher as being too rigorous. The trainer was seen as a strict educator and not so much as a friendly parent-type high school teacher, as they were used to during their classes.
Hiring high school teachers is currently undertaken between UPB and a variety of high schools, based on existing signed cooperation protocols and they do not imply additional costs. High school teachers are higher with a volunteer contract. This is a used protocol within UPB.
Point 13: It would be useful to summarise the main conclusions from the results and to discuss these alongside the limitations of the work.
Response 13: The limitations of the work were mentioned in the last section of the paper together with the main summary of the research – line 622 - 636.
Author Response File: Author Response.pdf
Reviewer 3 Report
Dear Authors
It seems to be a meaningful approach to the sustainability of the STEM program. I also read interestingly as a researcher in the field of STEM.
Here are some things you may need to supplement:
line 310
Table 2: Can this be seen as an increase compared to before applying the STEM program? Couldn't it be a high-level students?
If the pretest was not conducted, you should describe the limitations in the paper.
line 378-469
Conclusions
The suggestion for follow-up studies is rather weak. It is necessary to provide additional implications based on the study results.
Please describe the value of this study by comparing it with the previous studies.
line 488-494
Appendix A: Left align would be better.
line 495-520
Appendix B: 20 figures can be substituted to one table. Table increases visibility in this type of data.
line 522~
References: You'd better to include DOI to references and remove hyperlinks.
Author Response
Point 1: line 310
Table 2: Can this be seen as an increase compared to before applying the STEM program? Couldn't it be a high-level students? If the pretest was not conducted, you should describe the limitations in the paper.
Response 1: The study did not include a preliminary study of efficiency levels of the 20 components, as this would involve a highly complex evaluation matrix, due to the fact that the proposed STEM intensive programs are inter and multidisciplinary. Currently, based on the structure of the STEM intensive programs and on the obtained results of the present study, the authors are designing a framework to evaluate a compound efficiency of individual disciplines, as they are studied in the regular curriculum of secondary and tertiary education programs, such as: mathematics, physics, informatics, 3D printing, CAD etc. In the future, this will be the basis of a comparison evaluation in performance increase.
As suggested, this was explained in the limitation section of the paper – line 625
Point 2: line 378-469 – Conclusions
The suggestion for follow-up studies is rather weak. It is necessary to provide additional implications based on the study results. Please describe the value of this study by comparing it with the previous studies.
Response 2: Additional details about implications and added value of the study was inserted in the last section of the paper, lines 601 - 621
Point 3: line 488-494
Appendix A: Left align would be better.
Response 3: We implemented the suggested change.
Point 4: line 495-520
Appendix B: 20 figures can be substituted to one table. Table increases visibility in
this type of data.
Response 4: The authors consider that there is no significant advantage if the figures are placed inside a table, especially that the figures do not have individual captures. If the reviewer considers this as a mandatory change, we will implement it at his further suggestions.
Point 5: line 522
References: You'd better to include DOI to references and remove hyperlinks.
Response 5: As suggested by the reviewer, DOI was included instead of the hyperlinks, wherever it was available.
Author Response File: Author Response.pdf
