Evaluation of the Implementation of Project-Based-Learning in Engineering Programs: A Review of the Literature

Abstract

Project-Based Learning (PBL), as an experiential methodology, improves learning outcomes and competencies (technical and non-technical) in engineering students. The Conceive–Design–Implement–Operate (CDIO) approach, adopted globally in engineering education, is based on PBL but expands the curriculum framework. Developed by MIT and the Royal Institute of Technology (KTH) in Sweden, CDIO focuses on the entire life cycle of engineering projects to train engineers so that they are capable of applying knowledge in real-life situations. Integrating CDIO and PBL into engineering curricula requires changes in teaching methodologies, teacher training and workspaces. The literature has explored their combination, highlighting shared values and mutual reinforcements. An assessment model is crucial for implementing PBL and evidencing improvement in student and course skills. Only through assessment and the cycle of continuous improvement will the adoption of PBL in engineering programs be advanced. A systematic review of the literature is proposed to identify effective methods in the evaluation of educational programs based on PBL, analyzing related research areas and evaluations according to the CDIO approach.

1. Introduction

Project-based learning (PBL) is an educational method that emphasizes the development of higher-order thinking skills such as analysis, synthesis, and evaluation. This method of instruction facilitates deeper learning by engaging students in real-world problem-solving scenarios, requiring them to apply, analyze, and evaluate information, which are higher-level cognitive processes according to Bloom’s taxonomy [1].

To implement the competency-based approach, it is necessary to modify teaching methodologies, which involve a process of research, implementation, and development of methodologies centered on project-based learning and experiential learning [2,3].

Since its inception, the EHEA has aimed to enhance the quality of instruction by adopting a competency-based approach that focuses on skills that are relevant to employability, including cross-cutting competencies.

The European Higher Education Area (EHEA) is a region of educational organization that was established in 1999 through the Bologna Process. Currently, 49 countries are part of this shared space, which was designed to adopt comparable standards for university degrees, thereby enhancing professional mobility across countries.

Numerous universities in Europe have implemented systematic methodologies in their engineering degrees at the undergraduate level. Some of the institutions that have made significant contributions to the state of the art include the KTH Royal Institute of Technology (Sweden), Aalborg University (Denmark), Maastricht University (The Netherlands), and the University of Twente (The Netherlands).

In 2000, the CDIO (Conceive–Design–Implement–Operate) initiative emerged as a unified approach to adapting engineering programs to the new educational framework. This initiative was the result of a collaboration between the Massachusetts Institute of Technology (MIT) in the United States and the Royal Swedish Institute of Technology (KTH). The CDIO framework was designed to enhance the quality of engineering programs and prepare students for 21st century engineers. The first curriculum was introduced in 2001, and the first meeting between collaborating entities took place in 2010 [4].

The CDIO approach focuses on the overall organization of engineering programs and aims to produce well-rounded engineers who can apply their knowledge to real-life situations. The adoption of CDIO involves a comprehensive review of engineering curricula, including changes in teaching methodologies through the adoption of problem-based learning (PBL) as the most widely used approach, the evaluation of learning processes, teacher training, and the provision of workspaces, among other elements.

Since 2000, the CDIO (Conceive, Design, Implement, and Operate) movement has emerged as an umbrella that allows for the systematization of an experiential learning methodology. The combination of Project-Based Learning (PBL) with the Conceive–Design–Implement–Operate (CDIO) approach has been explored in the academic literature, highlighting that both methodologies share fundamental values and can partially overlap as strategies for educational reform [5]. A model of effective learning that integrates PBL and CDIO suggests that the learning communities of both methodologies can benefit from one another. However, while PBL and CDIO may play compatible and mutually reinforcing roles, they are not identical in their application and change strategies [5]. For example, the CDIO approach has a broader scope that includes program and standards perspectives, whereas PBL focuses on the learning approach within individual courses [6]. Furthermore, the integration of PBL into the CDIO curriculum has been applied in specific contexts, suggesting that a combination of both methodologies can be adapted to different disciplines and educational contexts.

An international comparative study of the CDIO curriculum indicates that it is well aligned with the ABET criteria and several international industrial company competence lists, such as Swedish requirements, EURO-ACE, and EURO-INF (European accreditations).

The integration of the principles and practices of project-based learning (PBL) with the CDIO approach can be achieved through their combination, considering the strengths and scope of each. The literature suggests that this combination can enhance quality and learning outcomes in engineering education, provided that careful implementation and consideration are given to the specific context of each educational program [5,6,7,8].

Since 2000, the initiative has experienced exponential growth, with over 120 engineering schools implementing CDIO standards in their undergraduate and graduate programs. The CDIO approach is grounded in the complete life cycle of engineering projects and aims to train integral engineers who can apply their knowledge to real-world situations. The implementation of CDIO requires a comprehensive review of engineering curricula, including modifications in teaching methodologies, evaluation of the learning process, training of instructors, and the provision of workspaces. It can be said that the CDIO Initiative is rapidly gaining momentum as an innovative educational framework for engineering education worldwide.

The proposed systematic review aims to identify the most effective methods for evaluating the implementation of educational programs based on a project-based learning methodology using the CDIO approach. This review provides a comprehensive analysis that identifies areas of research related to the evaluation of programmatic processes in engineering education according to currently defined standards. While a systematic review of the CDIO library was conducted in [9], it only covered the years from 2010 to 2020; therefore, it is necessary to update this review to cover 2021, 2022, and 2023.

The studies included in [9] were categorized into three types: CDIO Progress, which refers to policy or curriculum developments to update the standards defined in version 3.0; CDIO implementation, which refers to practical designs of adaptation to CDIO standards and is descriptive in nature; and Educational Research in Engineering, which refers to novel analyses of educational research.

However, the limitations of this study are that it has not been updated to 2024 and only includes studies from the CDIO library without integrating documents from other scientific databases

2. Objectives

The primary goal of this article is to undertake a comprehensive review of the scientific literature to identify studies that incorporate an assessment of the implementation of Project-Based Learning (PBL) methodology in engineering programs. Given that the CDIO philosophy, which is based on the four phases of the Conceive–Design–Implement–Operate project, incorporates the PBL methodology as a key component of program implementation, this review includes CDIO in the search criteria.

As with the previous objectives, the following were considered:

▪

Updating the most recent literature review of the CDIO library, including the proceedings of the 2021, 2022, and 2023 annual conferences (<https://www.cdio.org/> accessed on 1 April 2024), identifying as many empirical studies, reviews, and theoretical articles as possible that address the joint implementation of CDIO and PBL in engineering education.

▪

This search was expanded to include documents published in the Web of Science and Scopus databases.

▪

Identifying possible fields and educational research in engineering that have not been addressed or inadequately explored to reproduce them in any other context.

Locating possible evidence of program evaluation and whether there is a generalizable procedure for assessing any program of study

3. Methodology

A systematic literature review is a vital instrument in scientific and academic research, enabling the synthesis and organization of existing knowledge on a particular topic, identification of emerging trends and patterns, and provision of a solid foundation for informed decision making.

The PRISMA 2020 protocol [10] enhances transparency and accessibility in the retrieval, filtering, and categorization of study documents. This review comprises the research question, the eligibility and exclusion criteria for information source extraction, the obtained outcomes, and a discussion of the findings.

The methodology of the present systematic review focused on the assessment of the implementation of PBL and the CDIO approach in an engineering program, and it was guided by the following research question: “Is there any model to evaluate methods in the implementation of PBL in the literature review?” Three sources of information were selected to locate program evaluation studies:

▪

The SCOPUS Database;

▪

Web of Science’s Database;

▪

The CDIO library (www.cdio.org).

The CDIO library systematically reviews publications from the CDIO knowledge library (2010–2020) and classifies them according to [9] to provide an overview of trends and recommendations for future research up to the most recent conference proceedings of 2021, 2022, and 2023. The library utilizes a classification system previously established by the authors. The articles obtained in the last few years have been categorized into three main categories, as proposed by O‘Connor: (1) advances in CDIO that include curriculum policies for standards development, (2) CDIO implementation, including practical designs and start-ups, mainly descriptive and concrete; and (3) research in engineering education in which new developments are analyzed based on evidence.

From these, those that are not classified in the type of engineering education research are excluded.

A search was performed in SCOPUS and WoS using the following criteria.

Primary studies, systematic reviews, and meta-analyses reporting empirical results on the joint implementation of CDIO and PBL in engineering education programs will be included, and articles published in English or Spanish between 2010 and 2023 will be considered. Editorials, letters to the editor, and studies that do not specifically address CDIO and PBL integration were excluded (

Table 1. Inclusion and exclusion criteria.

Inclusion	Exclusion
Project-based learning	No problem-based learning No inglés o español No primary education No secondary education
CDIO
PBL
Engineering education
Higher education
Between 2010 a 2023
Opened access
Spanish and English

).

▪

All studies had to be based on project-based learning and/or CDIO standards. Problem-based learning was excluded from this study.

▪

Only in higher education, specifically in engineering undergraduate or graduate programs.

▪

The timeframe was from 2010 to 2023.

The first search yielded 100 documents from WoS, 140 from Scopus, and 224 from the CDIO library.

The present search incorporates an additional criterion, that of “evaluation.” This encompasses all documents that discuss the assessment of programs, students, or competencies. Following the second filtering process, 74 texts were selected. These texts were then classified into four distinct categories according to the nature of the evaluation identified in the document summary, and the data were compiled in an Excel table for further analysis.

I.

Evaluation of a program: Contains criteria for evaluating the implementation of the PBL methodology and/or is aligned with CDIO standards,

II.

Assessment of student competencies: Contains criteria for assessing the level of student competency acquisition after learning in a PBL and/or CDIO context.

III.

Evaluation of a teaching methodology: Contains an evaluation of the implementation of a new learning/teaching tool which, within PBL, incorporates some novel experiential learning techniques.

IV.

Others: Content that appears with the search criteria but, when reading the content, does not match what is being searched for.

From this filter, the number of documents according to typology is as follows: Type I, 23; Type II, 30; Type III, I9 and Type IV, 11. One article is excluded from Type 1 because it is not open access. The total number of articles considered for this review are the 23 resulting from Type 1 corresponding to “Program evaluation”.

The filtering and classification process was carried out by three independent reviewers in two phases: the initial review of titles and abstracts and a full-text review. In the event of disagreement, a consensus was sought. Data on study characteristics, participants, intervention, comparison, and results were extracted using a shared Excel sheet by all three reviewers. A flow diagram illustrating the filtering and selection processes is shown in Figure 1.

4. Synthesis of the Results

From all the documents read and classified, in order to answer the research question that is the subject of this review, how can we evaluate the implementation of PBL and the CDIO approach in an engineering program? The results of interest are those related to the evaluation of the implementation of a program and not to the evaluation of the competencies of the teaching agents, although this evaluation is closely related to the final result of evaluating a program. The suitability of a course depends on the achievement of learning outcomes, the acquisition of competencies, the degree of satisfaction of students and teachers, and the professional environment in which students develop at the end of their studies.

Figure 2 shows the percentage of each of the four types into which the documents are classified. The number of documents belonging to Type I accounts for 32% of the total final documents.

The following conclusions were drawn from the documents that analyzed how to evaluate the implementation of a course based on the PBL methodology and CDIO approach:

4.1. The Applicability of Incorporating the CDIO Approach in Evaluating the Project-Based Learning (PBL) Methodology

Several studies have merged the CDIO approach with the evaluation of PBL methodology in their curricula. These studies compare the criteria of the CDIO proposal with other criteria for the accreditation of engineering programs, such as European standards for the accreditation of engineering programs [7]. The CDIO curriculum outlines the competencies that an engineering student should acquire and provides an evaluation rubric aligned with the PBL teaching-learning methodology. In fact, the CDIO standard related to learning methodology asserts that it is imperative to adapt the curricula to an experiential learning methodology, such as PBL, to achieve the competencies needed for 21st century engineers [5].

4.2. The Need for Change in the Curricular Model of Engineering Students

The majority of the examined documents emphasize the necessity of updating the education, curricula, and teaching methodologies for engineering students to meet the evolving demands of the professional world. It is crucial to provide engineers with a comprehensive training program that encompasses both technical and cross-disciplinary skills to enhance their higher-order-thinking abilities in accordance with Bloom’s taxonomy [11,12,13,14].

The lack of alignment between the outcomes of engineering student training and the practical requirements of the professional environment is a critical issue. To bridge this gap, it is vital to align educational curricula with genuine demands of the industry.

To attain these outcomes, it is crucial to shift the teaching methodology toward active learning, transitioning from a teacher-centered approach in which the teacher is accountable for disseminating knowledge to a student-centered approach in which the student assumes the central position and the teacher acts as a facilitator or catalyst, encouraging students to question their assumptions and explore alternative solutions. This approach enhances student motivation, fosters autonomy, and bolsters self-assurance [15].

4.3. The Importance of the Project-Based Learning Methodology

It is widely believed within the educational community that the Project-Based Learning (PBL) methodology is the most effective approach for achieving necessary changes in engineering education. Studies have shown that incorporating the PBL methodology leads to improved technical competencies and the development of skills essential for an engineer, such as teamwork, oral and written communication, and complex problem-solving-abilities [16,17]. Furthermore, PBL fosters interpersonal competencies, such as time management and adaptability, in multicultural environments. These improvements in competencies are supported by competency assessment studies in courses that adopt the PBL methodology [14].

It is essential that the projects used in the PBL methodology are team projects that serve as a means to solve real-world problems in a professional environment. These projects should be interdisciplinary, allowing students to understand the relationships between different subjects, and should incorporate applicable technology that aligns with future professional life. The integration of an integrative project into courses has been shown to significantly improve learning outcomes, technical and non-technical skill acquisition, and competency development. For a project to be effective, it must integrate several disciplines and approach real problems from a professional perspective. If students do not perceive the usefulness of the project, demotivation may occur. The relationship between universities and companies is crucial in this regard, and designing collaborative projects between the two can improve students’ sense of belonging to the professional world and increase motivation [14,18].

4.4. Integration of the PBL Methodology in Curricula

Several approaches for incorporating problem-based learning (PBL) into curricula have been documented in the literature, including limited pilot programs that introduce an integrated project in a single discipline to the comprehensive implementation of PBL across the entire undergraduate or graduate engineering curriculum [15].

The specific implementation approach varies from institution to institution. While proponents of PBL generally highlight its benefits, such as increased student engagement and improved critical thinking skills, various challenges have also been reported, including increased workload for both students and faculty, larger group sizes than those recommended for PBL, and the need for specialized training or institutional commitment to facilitate the transition to PBL [15].

It is often recommended to implement PBL more gradually, starting with prescriptive approaches in early courses to reduce uncertainty for new students and transitioning to more open-ended PBL projects in later courses [15].

Some studies have used action research as a methodology for planning, executing, and analyzing results each year and have proposed improvement plans that address identified issues in subsequent years [19].

4.5. The Importance of Including Stakeholders

Most of the analyzed articles talk about the importance of including the requirements of all interested parties in the process so that the implementation is successful. An adequate curricular adaptation is not possible if all parties interested in the program are not taken into account, that is, students, teachers and future employees, as well as the needs of the industry [5,6].

4.6. Assessing Students’ Implementation of PBL

Student and course evaluations are two fundamental types of evaluations in an academic setting [1]. Student evaluation consists of self-evaluation, tests, exams, and peer evaluation, whereas course evaluation is concerned with the satisfaction evaluation of the stakeholders involved in the process [20].

Various methods have been suggested for assessing students, including self-evaluation, exams, and peer evaluation. Peer evaluation is particularly noteworthy because it encourages students to actively participate in the evaluation process, especially when working in groups. To improve student assessments, it is crucial to establish and communicate the rubrics used to evaluate students’ work. A well-designed rubric increases the likelihood of better student assessments.

One significant aspect of evaluating Project-Based Learning (PBL) is the implementation of a continuous evaluation. Since the development of the project is part of a continuous learning process throughout the course, it is essential to evaluate students at various stages of the course rather than just at the end. To achieve this, continuous feedback from the teacher is necessary as they must motivate and support students in discovering their own knowledge.

A combination of multiple assessment methods has been shown to substantially improve students’ outcomes. Some studies suggest creating an assessment portfolio that includes a variety of methods such as peer assessment, feedback, self-assessments, online assessments, and personal interviews. Using a variety of assessment methods provides a more objective method for grading students [20].

4.7. On the Evaluation of Courses in the Implementation of PBL

Several factors must be considered to assess the effectiveness of courses that utilize the problem-based learning (PBL) approach. One crucial aspect is the correlation between student evaluation results and course evaluation outcomes. A program is considered successful when students are adequately prepared for both their personal and professional development.

However, course evaluation depends on the satisfaction of all parties involved in the teaching process. It is not sufficient for students to perform better; all participants must evaluate different aspects of PBL implementation in the program to identify both positive and negative aspects and implement improvements [20,21].

Evaluation of the suitability of the methodology depends on a reliable data collection tool [22]. Data was collected through satisfaction surveys, evaluation questionnaires, and interviews with all the participants. Different methods can be used to gather data, such as digital tools, online questioning, face-to-face conversations, and direct feedback from the classroom. The key lies in asking the right questions and conducting reliable data analyses [23].

5. Conclusions

The importance of determining the purpose of an evaluation model should not be overlooked, as it is essential in directing attention to pertinent inquiries. Evaluating entails uncovering the value of a given action. To encourage the involvement of stakeholders, it is crucial to incorporate questions that can be answered and acted upon, such as what to investigate, how to conduct the investigation, when to participate, and how to apply the findings.

The aim was to obtain evidence of students’ proficiency in knowledge, attitudes, and skills through assessment mechanisms that guarantee adequate academic competence.

This literature review aimed to investigate the methodology and implementation of project-based learning (PBL) in engineering programs. A preliminary finding indicated that the CDIO approach is complementary to and extends the implementation of PBL and therefore should be taken into consideration in future research. The CDIO framework provides a structured curriculum for engineering education that can be used to adjust program curricula according to current needs, and its standards can serve as a basis for evaluating courses.

PBL is an effective methodology for enhancing learning outcomes and technical and non-technical competencies in engineering students. Experiential learning, which integrates theory and practice, is crucial in engineering disciplines. PBL is considered to be the best option for designing integrated projects that solve real problems and simulate professional environments.

In the evaluation of PBL implementation in courses, challenges were encountered that, although not insurmountable, required further investigation. It is necessary to develop a comprehensive evaluation model that incorporates data from all stakeholders involved in the process and utilizes a valid tool for processing such data. The results of this literature review present various evaluation models that employ different tools, all of which are valid in the studied context.

The CDIO website (www.cdio.org) offers self-assessment rubrics for institutions to evaluate compliance with each standard in five phases. These rubrics assess the program’s alignment with the standards, which include PBL implementation, but do not specify the evidence required to evaluate a program in an objective, reproducible, and procedural manner.

This review highlights the need for a globally applicable and reproducible criteria-based evaluation model for programs employing the problem-based learning (PBL) methodology. The authors suggested that an evidence-based evaluation model for engineering programs would be highly effective in the future.