Evaluation of the Implementation of Project-Based-Learning in Engineering Programs: A Review of the Literature

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

Thank you for the opportunity to review this manuscript. The manuscript aims to identify effective methods in the evaluation of educational programs based on PBL, analysing related research areas and evaluations according to the CDIO approach. Our review is presented as follows:

Introduction

The first and second paragraphs introduce EHEA and its aim to enhance the quality of education by adopting a competency-based approach. Then the third paragraph continues with discussing competency-based approach and how PBL fits into this. In our opinion, the introduction can start with PBL and how PBL can enhance the quality of engineering education to align with the title then later link it to the aim of EHEA.

Methodology

Research question-line 140 and 141: "How can we evaluate the implementation of PBL in an  engineering program?" We suggest that this RQ is modified to be consistent with the aim of the manuscript “to identify effective methods in the evaluation…” , e.g. what methods are effective in the evaluation….”  The authors should also include what constitutes effective methods. Is there any specific measure for ‘effective methods’ they include when filtering the literature for the systematic review?

In Figure 1, the bottom right box contains the  terms ‘tipo I,’ tipo II,’ and so on. Should ‘tipo’ be corrected to ‘type’?

 

Figure 1 states that 39 studies were included in the review, however, only 23 sources are listed in the references. Could you clarify the discrepancy?

Result

We suggest the authors check the result section for overall consistency between the title, aim, result, and conclusion.

In the filtering criteria, the authors mentioned that they updated with references from 2020 to 2023, however, in the result section, reference [5] is from 2014, and reference [7] is from 2011. In addition, reference [19] (line 298) is from 2017.

Lines 239 and 240 read: “This finding suggests that the CDIO proposal is a significant and comprehensive reference for defining and evaluating engineering careers.” This does not align with the heading (line 229 and 230): “The applicability of incorporating the CDIO approach in evaluating the Project Based Learning (PBL) methodology”

 

Conclusion

The conclusions are not consistent with the result. For example, lines 343 to 345 read: To encourage the involvement of stakeholders, it is crucial to incorporate questions that can be answered and acted upon, such as what to investigate, how to conduct the investigation, when to participate, and how to apply the findings. We could not find paragraphs in section 4 (result section) that discuss encouraging stakeholders’ involvement and investigation as stated in lines 345 to 345.

References

Please check the reference numbering, for example between line 69 and 82 in the manuscript, after reference [6] reference [7] is missing and the number jumps to [8]. Reference [7] however can be found in line 234 in the manuscript.

A reference is listed twice as reference number [9] and number [11]. 

 

Manuscript formatting, Tables, and Figures

Inconsistencies in paragraph spacing, e.g. line spacing of paragraph 1 and paragraph 2 in the introduction section are visibly different.

The Table under line 168 is not numbered and labeled.

Figure 2 label is own source, this should be changed to reflect what the figure is showing.

The legend in Figure 2 reads tipo, should this be type?

When referring to figures, there is an inconsistency (line 198: Fig 1 while in line 211: Figure 2).

Inconsistency in section numbering, for example after section 4.3 is section 1.4 and section 4.5. Should section 1.4 be 4.4?

Inconsistency in formatting of section headings, for example, section 3 is formatted bold but not section 4.

 

Line 126 should it start with a bullet?

Author Response

Dear reviewers: thank you very much for your suggestions; they are all very appropriate and I have made the changes you requested. I hope that it is more complete now. Once again, thank you very much for your work, which has been very helpful to me. Best regards.

Reviewer 2 Report

Comments and Suggestions for Authors

This paper presents a carefully performed literature review about the use of PBL within the CDIO context, updating CDIO reviews from a couple of years ago. The work leads the authors to conclude that PBL is a valuable tool to be integrated in CDI approaches. The authors point to the need for better evaluation models for comparisons between different implementations.

Author Response

Dear reviewers: thank you very much for your suggestions; they are all very appropriate and I have made the changes you requested. I hope that it is more complete now. Once again, thank you very much for your work, which has been very helpful to me. Best regards.

Round 2

Reviewer 1 Report

Comments and Suggestions for Authors

Thank you for the opportunity to review the revised manuscript. While we appreciate the authors’ efforts to address previous feedback, several comments remain as follows:

Methodology

Lines 204-207 read: “From this filter, the number of documents according to typology is as follows: Type I, 23; type II, 30; Type II, I9 and Type IV, 11. One article is excluded from Type 1 because it is not open access. The total number of articles considered for this review are the 23 resulting from type 1, corresponding to "Program evaluation". “

·       The capitalization of “Type” should be consistent throughout, please use either “Type” or “type” consistently.

·       “Type II,19”  should this be “Type III, 19”?

·       The total number of documents of Type I to Type IV sums to 83, however, in Figure 1, the number of articles to classify is 73. Please clarify this discrepancy.

·       The sentence “One article is excluded from type 1 because it is not open access.” does not align with Figure 1, where the second box from the bottom on the right reads: Excluded (n=0). If one article is excluded from Type 1, the remaining articles would total 22, please clarify this point.

In the previous review we pointed out that the timeframe of the articles was 2020-2023, however, there are references in the previous manuscript  that were published outside this range. The current revised manuscript’s timeframe is revised to 2010-2023. However, reference [4], which is discussed in Section 4 (Synthesis of the Results) is dated 2007. This needs clarification.

Synthesis of the Results

Figure 2 currently lacks labels and a legend to make it clear for the readers what each color in the pie chart represents in terms of document type.

The authors write: “Own source Figure 2 shows the % for each of the four types into which the documents are classified. Type I are 32% of the total. “

·       It would be more appropriate to write: “Figure 2 shows……”  without “ own source” and “shows the percentage” instead of “shows the %”

·       The sentence in line 261: "Type I are 32% of the total"  appears incomplete". A suggested revision could be: “The number of documents belonging to Type I accounts for 32% of the total final documents. Documents belonging to Type II account for ….% and so forth.

·       Please replace “%” in Figure 2's label with “percentage.”

·       The count of the references discussed in Section 4 is less than 23 (or 22 if one is excluded), please check again.

References

References [20] and [23] are currently missing publication dates and article types. Please provide this information.

General comments

Please check for any inconsistencies and typographical errors throughout the articles, for example:

·       Numerical consistencies as outlined above (23 or 22, 73 or 83?)

·       Inconsistencies in using capitalization of terms, such as “ Type” vs “type”, “CDIO (Conceive-Design-Implement-Operate)”-line 48 vs “CDIO (Conceive, design, implementation, and Operate)”-line 63.

·       Lines 213 and 214 read: A flow diagram illustrating the filtering and selection processes is shown in Fig. 1. However, the label of the figure is Figure 1 instead of Fig. 1. Please revise accordingly.

·       Typographical errors, for instance: Openes access in Table 1, this should be corrected to “open access.”

·       Please ensure that all tables and figures are appropriately cited. There is no reference to Table 1 in the manuscript.

We recommend using a language editing service to assist with inconsistencies, typographical errors as well as overall clarity.

 

Thank you for considering these points, we look forward to seeing the manuscript further refined.

Author Response

Dear Reviewer, Thank you very much for your comments, which are very timely. The changes made in the uploaded file are highlighted in color so that you can easily locate them. The responses to your comments are described in red. Thank you again for your corrections.

• Lines 204-207 read: “From this filter, the number of documents according to typology is as follows: Type I, 23; type II, 30; Type II, I9 and Type IV, 11. One article is excluded from Type 1 because it is not open access. The total number of articles considered for this review are the 23 resulting from type 1, corresponding to "Program evaluation". Modified
•  “Type II,19”  should this be “Type III, 19”? It´s correct. Modified

•   The total number of documents of Type I to Type IV sums to 83, however, in Figure 1, the number of articles to classify is 73. Please clarify this discrepancy. The sentence “One article is excluded from type 1 because it is not open access.” does not align with Figure 1, where the second box from the bottom on the right reads: Excluded (n=0). If one article is excluded from Type 1, the remaining articles would total 22, please clarify this point. You are absolutely right. It´s a mistake. Modified.

• In the previous review we pointed out that the timeframe of the articles was 2020-2023, however, there are references in the previous manuscript  that were published outside this range. The current revised manuscript’s timeframe is revised to 2010-2023. However, reference [4], which is discussed in Section 4 (Synthesis of the Results) is dated 2007. This needs clarification. The inclusion of this article is because it was talking about a model from the year 2001, but being strict with the time frame it is replaced by another one from that period which is also mentioned in the CDIO curriculum.

• Synthesis of the Results All typos mentioned in the review have been corrected.

•  The count of the references discussed in Section 4 is less than 23 (or 22 if one is excluded), please check again. Some of them are referenced before section 4. Do you think it is necessary to cite them again? If you think it's better, there's no problem.

Author Response File: Author Response.docx