Evaluation of STEAM Project-Based Learning (STEAM PBL) Instructional Designs from the STEM Practices Perspective

Abstract

Currently, there is a wide diversity of project-based learning instructional designs presented as “STEAM projects”. However, it is essential to evaluate if all these STEAM projects align their learning objectives and activities with the intended STEAM competences. This paper aims to characterize the impact of the STEAM educational approach through the analysis of contemporary STEAM projects implemented in five Spanish secondary schools from a curricular perspective based on STEM practices. A dataset comprising 46 secondary school STEAM projects implemented in Spain was evaluated using STEM project-based learningrubric, considering 21 evaluation criteria. The findings reveal an imbalance in the sophistication of STEAM projects concerning Science and Technology disciplinary-linked criteria and meta-disciplinary-linked criteria within this framework. These results enable the mapping and highlighting of the fact that not all STEAM projects equally serve their intended educational purposes or integrate all their features with the same level of sophistication. Curriculum organizations from different secondary school levels are also pointing out notable differences regarding how they address STEM competence. Acknowledging these differences and challenges in further initiatives of STEAM PBL instructional designs could support their design. By identifying areas of improvement, educators can optimize the impact of these projects on fostering STEAM competences among students.

1. Introduction

All across the world, there is a need to address environmental and social challenges, such as overpopulation, resource management, health, decrease of biodiversity or even the impact of new technological tools. These challenges are expected to increasingly gain importance and demand for a multi-layered response and a collective effort which will ask for informed decisions from the whole population [1]. Therefore, part of this challenge pertains to education, where many countries worldwide have already adapted their educational goals to focus on competency-based standards. In this context, STEM education emerges as an essential factor to enable decision making and forces us to rethink more suitable ways to be competent in the STEM disciplines in today’s society. Part of this rethinking process has already pointed out the need to adopt intersectional lenses to understand STEM from both affective and epistemic domains [2]. Only then can we make sense of the current low interest towards STEM from specific groups of people and identify the best ways to interconnect such disciplines for all to gain STEM literacy.

The following discussion on STEM integration encompasses issues such as which forms of knowledge and disciplinary practices should STEM be accounted for, as well as the best methods of integrating disciplines, through transdisciplinary, interdisciplinary, multidisciplinary approaches, and more [3]. The discussion has become even more complex with the addition of the “A” for Arts in STEAM, where controversy is raised around its understanding and its role [4].

In this still confusing scenario with different interpretations of STE(A)M, national and international plans for promoting STEM and STEAM education are deployed. In some contexts, these plans are normally turning into a wide diversity of methodological approaches that claim to address STEAM literacy in secondary education.

One of the available options to promote STEAM education is project-based learning (PBL), due to its characteristics that potentially facilitate 21st century skills that are essential for addressing the aforementioned challenges [5]. We believe that a clearer focus on how STEAM PBL becomes tangible in the day-to-day classroom is key to make sense of how teachers set and share expectations and reach decisions around activities, and find ways of assessing or strategies to unfold contents. Therefore, we refer to STEAM PBL instructional designs as comprising the entire set of resources established in the educational instructional units within the PBL pedagogical framework that serve to acquire STEAM competences. While there is increasing empirical research around effective examples of STEAM implementation through PBL instructional designs [6], there is still little literature that showcases how these different STEAM PBL instructional designs are displayed in practice.

Particularly, there is little research analysing if these STEAM PBL projects are good teaching and learning scenarios from the lenses of widely accepted frameworks in science education that place the focus on the practices and not only the products of science [7]. In a national context, this framework resonates with the ideas of Izquierdo-Aymerich and Adúriz-Bravo [8] that embrace the same philosophy and also make operative other key educational principles, such as (1) selecting core ideas of science, (2) context-based learning and (3) formative assessment.

For this purpose, this study focuses on a metropolitan region of Catalonia in Spain, which serves as a compelling case for the empirical development of STEAM PBL in secondary school classrooms. In Catalonia, STEAM education has gained prominence both at the policy and academic levels, coinciding with a broader innovation movement that embraces PBL as a preferred teaching and learning approach [9]. At a policy level, the pedagogical plan involves methodological features, such as context-based and student-centred pedagogies. These approaches enable critical decision making in a sustainable and inclusive manner while encompassing a formative assessment perspective [10]. The secondary schools in this region serve as pertinent cases for exploring how educators conceptualize STEAM and PBL through the creation of so-called ‘STEAM projects’.

To shed light on these matters, this paper aims to characterize the impact of the STEAM movement in contemporary STEAM PBL instructional designs implemented in five schools of Spain from the STEM practice-based framework. This general goal is further concretized into three sub-objectives:

• Identify the instructional design sophistication of 46 currently already implemented STEAM projects by utilizing a rubric instrument built from the STEM practice-based framework.
• Compare how the STEAM projects gathered from five secondary schools are characterized in their instructional design sophistication dimensions.
• Analyse the differences and similarities between schools that exclusively convey science curriculum through STEAM projects from those which also use other complementary methodological approaches.

In the following section, we delve into the theoretical framework from which this paper addresses the previous research questions and facilitates further discussion of the results.

2. Theoretical Framework

2.1. From STEM to STEAM Education

There are already different research contributions that have attempted to shed light on the conceptualization of STEM (Science, Technology, Engineering and Mathematics) [11], and STEAM [12]. While we hold a better understanding of the socio-political origins of the STEM acronym, today there is a rich conjunction of perspectives and motives for STEM education at different educational levels [13]. These motives may include the need for learning in more authentic situations, looking at the world from a set of different epistemological lenses, a more active and empowered citizenship, etc. [2,9]. A comparable analysis is conducted concerning STEAM, examining the rationale behind the integration of the Arts (A), and presenting different perspectives on balancing STEM versus Arts disciplines or seeking a more overarching educational term that emphasizes creativity [4].

As mentioned, the integration of these specific S-T-E-M disciplines is not arising from an initial academic reflection, which has raised concerns about what is truly new from it [14]. Notwithstanding, different authors have explored the epistemological foundations of these disciplines enabling different interpretations to integrate them [3,12,15]. This discussion has triggered the need of specifying when new instructional proposals are truly “integrated STEAM”, which preferably adopts interdisciplinary and transdisciplinary approaches of discipline integration [12].

The appearance of the A for “Arts” in STEAM has also placed the discussion into the Arts meaning and its role in conjunction with the rest of the STEM disciplines. From the current literature, Arts are normally related to a set of disciplines such as liberal arts (humanities, social studies, language) and visual and fine arts (also considering physical and musical disciplines) [16]. At the same time, this “A” can be strategically introduced to provide a better balance of disciplines or enhance certain skills and attitudes, such as creativity, critical thinking and innovative spirit [4,17].

From the ongoing reflection about STEM and STEAM, we align with those conceptualizations that are grounded on promoting STEM literacy for all students. From our perspective, this purpose entails two clear needs: developing students’ STEM competence, which demands a revaluation of STEAM practices from an epistemic reflection and developing students’ identities in a way that are caring and reconcile students’ selves with how they think STEM people are [2].

By understanding how learners relate and identify with STEM, better inclusive initiatives that allow students to bring their own prior experiences, practices and ways of being to the arena can be developed. This demands not only less stereotyped examples of STEM people, but also paying attention to other aspects such as self-efficacy, aspirations or capacities [18]. We suspect that these attempts of designing and enacting more inclusive and equitable approaches into the classroom can be also supported when these approaches are interwoven with the second aforementioned need. By shifting the focus and including more representative practices of STEM disciplines in the classroom, a more naturalized image of these disciplines can be delivered, which at the same time can provide good opportunities to transform the stereotyped and sometimes exclusive self-representations of STEM into an activity that is more inviting to share and discuss everyone’s ideas [19]. This second need also points out the need to grasp a better understanding of each discipline and how they can relate to each other to enhance STEM competences [20]. Within this framework, integrated STEAM education should be accountable for distinguishing core defining elements of every discipline, such as their goals and practices (which may allow for a multidisciplinary approach), from shared elements of different STEM disciplines, such as their problems, methods and values (which can be easier integrated in a more sophisticated manner) [21].

When specifically talking about the role of Arts, we find it particularly interesting to accentuate how the Arts conception can be connected with other approaches with a longer tradition in science education, such as Science, Technology and Society (STS) [22]. From the mentioned understandings of A in STEAM, the STS approach resonates with the need of integrating Arts as liberal arts, bringing the unavoidable social and humanistic lenses of every complex STEAM challenge. Despite this we adopt a broader perspective that acknowledges and embraces the diversity of forms and roles that Arts play in the STEAM instructional designs.

2.2. Project-Based Learning as a Means of STEAM Education

Among the diverse palette of methodological approaches to convey STEAM education (tinkering, design-based learning, inquiry-based learning, maker movement, etc.), PBL has been one of the preferred ones [6,15,16,23]. This alignment seems natural due to the overlapping interests of STEAM education and the overarching purposes from which PBL is normally justified. As an example, both STEAM and PBL educational approaches demand students to have an active role in their learning and their tasks by connecting them to the school community and tangible outcomes, which are normally referred attributes of PBL [24,25].

The literature regarding PBL in science education started in the 1990s [26,27], where the influence of the evolving trends enabled the conceptualization of the so-called project-based science (PBS). This PBS is conceptualized with 6 key features: (1) starting with a driving question with an implicit problem, (2) focusing on key learning goals, (3) engaging in scientific practices to answer the driving question, (4) participating in collaborative activities, (5) use of learning technologies to scaffold the practices of science, and (6) creation of a tangible product as an external representation of the class learning [28,29]. In this context, some authors have pointed out the complexity of finding a balance between the engagement in in-depth curricular contents and the practical effect of the project [30]. The author describes it as an “unresolved tension” that especially appears in “performance PBS” projects, which are defined as those whose principal activity is designing, within the sense of design-based science [31]. This performance PBS approach to convey the Science curriculum seems to be the most commonly found when looking at the literature about PBL within STEM education [32,33]. Even when other existing initiatives emphasize the role of inquiry in STEM PBL instructional designs [34], a more general conception is frequently adopted around inquiry, which may become distanced from the more scientific purpose of it [27]. In this scenario, further insight is necessary to understand how STEAM education may be dealing with these tensions that have been also reported in other science learning contexts [35,36].

3. STEM Practices as an Analytical Framework to Study STEAM PBL

In order to analyse STEAM projects, this paper opts for a perspective that enables insight from a coherent and operational framework that focuses on teaching and learning STEM practices in the same trend as Science and Engineering practices are already jointly presented [37].

We align with those frameworks that focus on the practices of STEM disciplines since they are widely supported by many researchers and educational policies [7] and resonate with the reflections from philosophical, psychological and historical reflections in the context of science education [38]. From this perspective, a cognitive model for STEM education can be developed in a similar way, problematizing how students engage in the different spheres of activity of Science [39] and Engineering [21]. More recent literature with the same focus on the practices also explores the synergies and boundaries between them, for example, between Science, Engineering and Mathematics [2] or between STEM and Arts [40]. Part of the endeavour of being STEM competent is therefore being knowledgeable about the epistemic objectives of each discipline. For example, the practice of ‘analysing and interpreting data’ can imply the mobilization of very different skills for different purposes. In Science, data analysis serves the purpose of improving descriptions and explanations of phenomena; for Engineering, it enables the identification of improvement points; and it allows for the verification of mathematical solutions in Mathematics [2].

In the context of this study, this practice-based framework aligns and is further operationalized by the School Scientific Activity framework, which also interprets that science classrooms must emulate the social, discursive and cognitive activities of scientists while the pursued goals, methods and final mental constructs should be tailored for the school settings and purposes [8,41]. We believe that the SSA framework provides extra insights into the STEM practice-based framework since it allows to grasp other key aspects for both scaffolding instructional designs and promoting adequate STEAM teaching and learning. More specifically, it provides operative guidelines to (a) address context-based proposals [42], focusing on the role of context towards a balance between the sense making of phenomena and finding personal relevance for the learner; (b) structure knowledge in core ideas in a coherent way with other proposals [7,43]; and (c) focus on formative assessment practices embedded in the school STEM activity [44].

4. Methods

This paper is part of a greater research project that is inspired by ethnographic approaches. Therefore, it is aimed to address and better understand teaching beliefs, practices and choices around the emerging use of STEAM PBL instructional designs in the context of a Spanish region (Catalonia). The research methodology is presented from an interpretative paradigm where both qualitative and quantitative data were used. Therefore, we conceptualize the term impact in an exploratory and qualitative manner, where the resulting sophistication levels in STEAM PBL instructional designs are informative of differential characteristics influenced by the STEAM approach we stand for.

4.1. Data Collection

In order to gather STEAM PBL instructional designs, normally referred to as simply “STEAM projects”, a selection of five high schools that design and implement STEAM and STEM projects was performed in convenience for this research. Secondary school centres were chosen for the following key features:

• Secondary school centres belong to a self-organized network of schools that discuss and develop curriculum materials with a PBL methodological approach.
• Secondary school centres are socially and culturally diverse and representative of the region they belong to.
• Every secondary school centre implements the PBL methodological approach in different pedagogical settings in terms of schedule structure, project duration, teacher expertise, integration with other methodologies, etc.

Contact with the different high school centres was maintained from 2017 to 2019. In this period, the authors had formal and informal interviews with different teachers involved in STEAM project design and participated in one of the school network meetings. These interactions with the participant schools allowed the authors to share the aim of this research and gather the required STEAM project data. In this process, participant schools showed a rich variety of ways to approach STEAM education. This diversity involved different ways of integrating STEAM disciplines and different ways of understanding the “A” in STEAM. Further information of the characteristics of every school are available in Appendix A.

Collecting STEAM PBL instructional designs from schools entailed gathering data in various formats, including virtual curriculum materials, teaching guides, curriculum planning documents, assessment tools, webpages, etc.

The criteria used to select STEAM PBL instructional designs were as follows:

• PBL instructional designs should be acknowledged as “STEAM projects” by the teachers who design and implement them. “STEM projects” were also included when “Arts” were explicitly integrated in the forms of plastic/musical arts, liberal arts and creativity skills.
• PBL instructional designs should incorporate operative elements of the PBL methodology and Science and Technology curricular standards.
• PBL instructional designs should have been tested at least once with 12–14-year-old learners.

Since this research is aimed to embrace a rich perspective around STEAM education, as authors, we did not limit the understanding of STEAM to a constraining vision or conceptualization. At the same time, we did not exclude STEAM projects that also embraced other compatible educational approaches (STS, IBSE, etc.). From the 49 STEAM projects gathered, 46 followed the required criteria for this study (see Appendix C). The titles of these STEAM projects provide insights into a diverse range of learning contexts and outcomes. They span from constructing a car for a competition (seen in the ‘Electric Car Race’ project) to engaging in a climate-focused conference (‘Air Water Congress’ or the ‘Pollution, Health, and Environment’ project).

4.2. Research Instruments

In order to evaluate the impact of the STEAM movement on real STEAM projects from the perspective of the SSA, we used theSTEM PBL rubric instrument [36] due to the following reasons:

• The STEM PBL rubricis already published and its content validated in an evaluative process.
• The design and validation process was part of the same research within a greater research project, providing to the rubric a strong theoretical foundation in the use of the PBL methodological approach in Science and Technology.
• Participants in rubric design were Science and Technology teachers and science education academics that were both informed about the SSA framework and the socio-educational context where the study was developed.
• The rubric offers a thorough analysis of STEAM projects since it considers 21 rubric criteria that enable evaluation of 6 different fundamental aspects of PBL in Science and Technology education. Therefore, a wider perspective can be offered as compared to other available rubrics [45].

Despite the benefits of this rubric, it should also be outlined that the SSA framework has traditionally been focused on Science and Technology education, and requires further exploration of mathematics aspects, which are not considered in this study.

The STEM PBL rubric is distinguished by a collection of 21 criteria (as listed in Appendix B) organized under 6 fundamental aspects: (a) Project Objectives; (b) Action; (c) Context; (d) Contents; (e) Science and Engineering Practices; (f) Assessment, ICT, and Collaboration. The authors also pointed out that 18 of these criteria can be divided between those that serve discipline-specific objectives of Science and Technology education (disciplinary criteria) and those that serve non-discipline-specific objectives of education, so-called meta-disciplinary criteria. The other three rubric criteria integrate both types of purposes.

Within each criterion, a breakdown of four distinct levels of sophistication was provided showing specific characteristics in a short description. These levels collectively delineate the path towards design enhancement. The term “sophistication” was used to describe the complexity of instructional design across each criterion.

4.3. Data Analysis

Data analysis implied two phases using qualitative and quantitative data. In the first phase, qualitative data were used from the curriculum materials of STEAM PBL instructional designs. Every facilitated document was coded per centre and STEAM project. A descriptive summary for the distinctive elements of every STEAM project was created to facilitate an overview and further evaluation. To showcase the relevance of this task, certain STEAM project summaries described which types of activities were more frequent than others and thus facilitated further interpretation of the significance of these activities in each STEAM project. In other cases, a STEAM project summary was relevant when it facilitated the interpretation of a potential mismatch between the explicit general learning outcomes and the activities the learners were involved in.

In this phase, every STEAM project was later interpreted by means of the STEM PBL rubric. This task involved analysing the STEAM project primary and secondary (summaries) documents and linking them to one of the four available descriptors for each rubric criterion (Figure 1). For every criterion, an explicit justification for the selected sophistication level was provided.

As an illustration, an excerpt from the STEM PBL rubric is included to demonstrate the progression in complexity concerning how STEAM projects engage with conceptual contents (

Table 1. An excerpt from the STEM PBL rubric, showcasing the four potential levels of sophistication used to evaluate the depth of scientific conceptual content within STEAM projects.

Rubric Criterion	Level 1	Level 2	Level 3	Level 4
Deepening on the conceptual contents	Eventually, descriptive contents in the form of information or data are incorporated. They are presented disconnected from each other.	Contents are selected that allow the description and identification of specific phenomena that are easily interpretable. It mainly involves simple cause-and-effect relationships.	Key ideas are selected and organized that appear recurrently throughout the project and are specifically developed at certain moments of the project.	Key ideas are selected and organized that appear recurrently and are progressively developed over different moments to build a theoretical model that allows explaining a wide range of phenomena.

). STEAM projects that introduce fragmented or unconnected information are categorized as level 1 in terms of sophistication. In contrast, those STEAM projects structured to encounter and systematically unfold complex scientific core concepts, such as Energy, Chemical Change, or Life Forms, are affiliated with higher sophistication levels.

This phase resulted in a dataset of 46 STEAM projects with 21 sophistication levels linked to each one (ranging from 1 to 4). To warrant reliability of this first-phase analysis, two external researchers analysed 9 random STEAM projects using the rubric. Most rubric sophistication levels were identically interpreted and those which were not identical were further discussed and agreed on their interpretation for the rest of the STEAM projects.

The second quantitative phase implied the use of descriptive statistics to analyse the generated dataset. In order to answer the first research question, the arithmetic means, as well as the standard deviations, were calculated for every STEM PBL rubric criteria. From this information, every criterion mean was discussed with the support of the STEM PBL rubric sophistication level descriptions.

In order to answer the second research question, data from the rubric criteria were pooled for every STEAM project by adding every resulting sophistication level score (from 1 to 4). In this case, only 18 criteria were used to create a STEAM project score. The reason for this reduction of rubric criteria was the need to select just those criteria that were clearly linked to disciplinary and meta-disciplinary aspects of STEAM PBL instructional designs. This data representation method enabled a further interpretation of the STEAM projects’ sophistication for these two particular sets of criteria. The scoring system was, therefore, providing two different scores for every STEAM project with a potential maximum score of 32 points for disciplinary criteria and 40 points for meta-disciplinary criteria. Results were represented in a two-dimensional map. To delve deeper into the findings, the meta-disciplinary criteria scores, primarily indicating PBL characteristics and objectives, were used as a practical means to assess how STEAM projects facilitate 21st-century skills. Likewise, the disciplinary criteria were employed as a practical method to evaluate scientific competence. This correlation is endorsed by the creators of the STEM PBL rubric [36].

5. Results

The results are presented in three subsections in accordance with the research objectives. Section 5.1 identifies the instructional design sophistication of 46 already-implemented STEAM projects. Section 5.2 shows a comparison between the design sophistication of STEAM projects in five secondary schools based on disciplinary and multidisciplinary criteria. Section 5.3 provides an analysis of the differences and similarities among schools that convey science curriculum through STEAM projects in different ways.

5.1. STEAM PBL Instructional Design Sophistication: A Breakdown of 21 Criteria across 46 STEAM Projects

In this study, 46 STEM projects were analysed by using a rubric instrument that allowed to associate an instructional design score up to 84 points (21 criteria with 4 progression levels) to every project. The average scoring of this set of projects was 45 points with a standard deviation of 8.78, which represents the (53% of the maximum scoring). This result shows the challenge to design highly sophisticated projects according to all the rubric criteria.

In Figure 2, a detailed picture of the average scoring of every criterion for the 46 STEAM projects can be read and further interpreted. As a general trend, dispersion bars inform of a relatively high diversity of sophistication levels for most of the criteria, which points out important differences in those projects acknowledged as STEAM. Hereafter, an analysis of the results is presented for every fundamental aspect assessed, linking each criterion to a certain sophistication level of the rubric (levels 1–4).

Rubric criteria linked to the fundamental aspect of “project objectives”, which include curricular goals, didactic goals and project goals, receive a score that ranges between sophistication levels 2 and 4. According to the rubric, this implies that STEAM projects are designed expressing competence-based goals, with a clear intention to develop 21st century skills, such as creativity, critical thinking and collaboration (description of level 2 of the rubric). Some STEAM projects may also show higher sophistication in their goals, explicitly including scientific literacy goals. From a didactic goal criterion, this involves introducing activities that mainly enable knowledge transfer into practice, which align to some extent to the project challenge (levels 3 and 4).

The criteria linked to the fundamental aspect of “contents” (conceptual, procedural, attitudes and integration of contents) largely vary among STEAM projects. The STEAM projects that show low levels for these criteria are mainly selecting descriptive concepts (level 1) or introducing simple scientific concept relations (level 2). Procedural contents normally imply showing (level 1) or making use (level 2) of certain rote procedures or prompted techniques. Values and attitudinal scientific contents normally appear in an implicit way (level 2). These values are sometimes more explicit (level 3) in specific activities that enable working sustainability values, health habits, etc. Contents from different STEAM subjects are normally presented in a multidisciplinary approach with clear divisions among disciplines (level 2) or just clearly integrate the contents and practices of one subject while the rest do not hold a relevant role (level 1).

Rubric criteria linked to “contextualization” evaluate scientific significance, relevance and authenticity. STEAM projects from this study normally involve natural and physical phenomena (level 2) that are sometimes scientifically interpreted (level 3 and 4). STEAM projects also involve learners in plausible situations of the real world (level 3) that respond diversely (levels from 1 to 3) to their personal and social interests.

Rubric criteria linked to the fundamental aspect of “action” inform about a general trend of STEAM projects that execute well developed actions (level 3), which, in some cases, are later evaluated (level 4). These actions are normally guided (level 1), but in some cases there is room for decision making by learners (levels 2–3). Action impact is also variable, from projects whose action is targeted to the classroom to actions that are aimed to involve the whole school community (level 3).

Rubric criteria linked to the “engagement in scientific and engineering practices” evaluate the scientific spheres of activity of modelling, inquiry and argumentation, as well as engineering practices. Argumentation skills are promoted in a broad sense of argumentation in the majority of STEAM projects (level 2). Inquiry practices are quite variable in their presence in STEAM projects but, on average, focus on data collection rather than promoting the use of evidence or the drawing of conclusions (level 2). Modelling practices are mainly focused on descriptive forms of presenting scientific knowledge (level 1) rather than on fostering more guiding interpretations of natural and physical phenomena.

Finally, the rubric criteria linked to the assessment and the use of ICT resources and collaboration show the following behaviours. Assessment design in STEAM projects is focused on co-assessment and formative assessment strategies that make use of fixed criteria facilitated by the teacher (level 2), where the final action has the biggest weight of the summative assessment (level 2). The use of ICT (Information and Communication Technology) resources is diverse, from a limited use to support the presentation of the final action (level 1) to more sophisticated uses to organize and communicate ideas (levels 2 and 3). Collaboration is regulated in a sophisticated manner that encompasses collaboration guiding instruments (level 2 and 3) and sometimes strategies to facilitate ideas exchange (level 4).

From a general perspective, Figure 2 is informative about the general trends in which STEAM projects tend to be sophisticated. Despite a high dispersion for the whole set of criteria, there is clear room for design improvement in fundamental aspects, such as “contents” and “S&E practices”. On the other hand, there is also a general trend to design STEAM projects that are sophisticated in the deployment of their actions, the ways to foster collaboration among learners and the way to present and to identify learning, didactic and project goals.

5.2. Distribution of STEAM Projects for Disciplinary and Multidisciplinary Criteria

The representation of the scores of the 46 STEAM projects in 2 dimensions allows us to better understand the diversity of STEAM proposals designed by schools.

From Figure 3, focusing the attention on the two axes, there is a clear tendency in STEAM projects to score better at multidisciplinary criteria (y axis). The cloud of projects is clearly displaced towards the upper part of the map, where no project scores less than 15 points out of 40 available. In contrast, STEAM projects spread all over x axis, showing a greater diversity of scores and therefore instructional design sophistication for disciplinary criteria. Furthermore, the map also shows that a small number of STEAM projects reach high scores in both of the two dimensions presented.

By looking at the map by schools, STEAM projects designed by different schools tend to scatter differently through the map, informing about more similar or different ways of designing STEAM projects within the same school centre. Schools 3, 4 and 5 are very representative of secondary schools that design STEAM projects very alike in their sophistication levels since they cluster in very specific regions of the map. Schools 4 and 5 are normally located in the upper left side of the graph, and encompass STEAM projects that are focused on multidisciplinary criteria but that are rather less sophisticated in S&T disciplinary criteria. In the case of School 3, the clustering is even more clear, since all their designed STEAM projects are in the central top right side of the map.

On the other side, Schools 1 and 2 show a more scattered design of STEAM projects, which are distributed all over the map, mainly across the x axis. This implies that they maintain a certain degree of sophistication for meta-disciplinary criteria (above 15/40 points) while showing a wide sophistication diversity for disciplinary criteria.

5.3. An Analysis of the Differences and Similarities between Schools That Differently Convey the Science Curriculum through STEAM Projects

The results in this section are divided between schools exclusively using STEAM projects for their Science curriculum and those employing additional complementary methodological approaches.

5.3.1. Secondary Schools That Exclusively Convey the Science Curriculum through STEAM Projects

Secondary schools that entirely convey their Science curriculum through STEAM projects have shown to perform differently on the design of their projects considering disciplinary and meta-disciplinary criteria. Schools 1 (n = 10) and 2 (n = 11) show a balanced profile between disciplinary and meta-disciplinary scoring (Figure 4). Despite this, dispersion of disciplinary criteria ranges from 30% to 70% of the maximum score for School 1. On the other hand, School 5 (n = 13) shows an unbalanced profile between disciplinary and meta-disciplinary scores in their projects and illustrates a tendency to foster meta-disciplinary criteria (56%) over disciplinary criteria (33%).

5.3.2. Secondary Schools That Partially Convey the Science Curriculum through STEAM Projects

From the five centres, Schools 3 and 4 develop their Science curriculum through different methodologies besides the use of STEAM projects. School 3 invests half of its available time for Science learning on developing STEAM projects. For the equivalent purpose, School 4 invests ⅓ of its available time to implement STEAM projects.

These secondary schools that partially convey their Science curriculum through STEAM projects have shown to perform differently on the design of their projects considering disciplinary and meta-disciplinary criteria. School 3 (n = 5) shows a balanced profile between disciplinary and meta-disciplinary scoring (Figure 5), being the one that shows the highest score average, although its number of STEAM projects is the lowest. On the other hand, School 4 (n = 8) shows an unbalanced profile between the disciplinary and meta-disciplinary scores in its STEAM projects. In this school, meta-disciplinary scores are comparable to other schools, while disciplinary scores vary considerably and obtain a low average score of 27% of their maximum, which is similar to School 5.

6. Discussion

6.1. Discussion of the Empirical Characterization of STEAM Projects

The resulting evaluation of the 46 STEAM projects for every criterion shows a picture where high dispersion among the evaluated criteria suggests that attempts to provide a general definition of STEAM projects in this context are troublesome. While the literature has provided different conceptualizations of PBL methodological approaches to introduce Science (such as STEM PBL, PBS, etc.), we wonder about which characteristics should be defining the features of any STEAM PBL instructional design. Data reported in this study suggest that the high dispersion among criteria could also be aligned with different forms of approaching STEAM education. This conjecture would fit into the recurring attempts in the literature to classify PBL into different types [30,46].

In this paper, we have chosen a basic distinction between disciplinary and meta-disciplinary criteria to provide more insights in this matter. Figure 2 shows that STEAM projects tend to show a better sophistication alignment for those meta-disciplinary criteria that mainly inform PBL features (PBL design elements, its pursuit for 21st century skills, etc.). On the other hand, larger dispersion was perceived in the data for the disciplinary criteria, which mainly inform about Science and Technology teaching sophistication of the STEAM projects. Therefore, it can be stated that issues that aim to find common ground in a generalization of a STEAM project, even for a relatively small sample as provided in this paper, are not focused around conceiving different ways of approaching PBL but different ways of approaching of teaching and learning Science and Technology. This resonates with the different approaches that normally influence STEAM PBL instructional designs, such as design-based science [31], inquiry in all their different conceptions [28,34,47] or simply focusing on teaching of 21st century skills in scientifically relevant contexts [48].

Moreover, subtle trends in the data identify potentially concerning aspects of how STEAM projects are designed. While most of the diversity in our data can be aligned with different conceptualizations of the learning of Science and Technology through PBL, the low sophistication levels in how STEAM projects may engage learners in Science and Engineering practices is not encouraged in any conceptualization [49]. Therefore, the already identified tensions between different “doing the project” and “learning the content” approaches [30] should be better explicated in this context concerning the mentioned STEM practices.

6.2. Discussion of Differences between Subsets of STEAM Projects by Schools

To discuss the results, we propose to divide the map of Figure 2 into its four sections that help to understand how different schools tend to distribute their STEAM projects. It can be argued that disciplinary criteria, through the evaluation of STEAM PBL instructional design, can be informative of how projects support the development of a scientific competence. In the same line, it could be argued that meta-disciplinary criteria are informative of how projects support the development of 21st century skills. Therefore, according to where a project is spotted on this map, we can infer its potential to foster a higher or lower scientific competence and 21st century skills.

STEAM projects scores by school and its implications for each can be organized in three different sets.

Schools 1 and 2 (Figure 3) convey their S&T curriculum through STEAM projects. They spread in the upper sections of the map (Figure 2), which is the zone where 21st century skills are better developed. On the other hand, some of their STEAM projects can be really fostering scientific competence through their designs. From the lenses of how these schools manage the tension, it can be discussed that they leverage them by shifting the focus on disciplinary criteria differently on each STEAM project. Despite this, we wonder in which situations scientific competence development represented by disciplinary criteria outperform meta-disciplinary criteria, signalling a clear focus on science engagement and understanding.

School 3 is partially conveying the S&T curriculum through STEAM PBL, which attempts to constantly balance between disciplinary and meta-disciplinary purposes. Despite being in the zone that can be promoting both types of competences ensuring a minimum design quality for all the STEAM projects, the lack of dispersion also quenches the possibility for delving deeper into specific criteria. Therefore, we wonder when STEAM PBL instructional designs should be truly aiming for a balance of these tensions in every STEAM project or a balance at a curriculum level that includes intentionally imbalanced STEAM projects in different situations. In this sense, we ignore if this focus in science instruction exists when using alternative methodological approaches.

Schools 4 and 5 show similar profiles, especially focusing on meta-disciplinary criteria and, therefore, in the left sections of the map (Figure 2), inform about their potential to develop 21st century skills. However, these schools show a different organization to convey the S&T curriculum. While School 5 conveys the S&T curriculum through PBL completely, School 4 does not. Therefore, without enough projects that bring the required sophistication to S&T contents, practices in these schools may challenge the development of scientific competence. On the other hand, School 4 may have some room for developing this competence conveyed through other methodological choices.

7. Conclusions

Based on the preceding discussion, several conclusions emerge regarding the current design of real STEAM PBL initiatives. Firstly, it is apparent that a standardized understanding of STEAM project design lacks a consensus within the context under study. This observation stems from the wide-ranging sophistication levels found in the analysed STEAM projects and echoes the need seen previously to categorize PBL approaches.

Secondly, the variations among STEAM projects predominantly centre around differing approaches to Science and Technology education rather than distinct methods of crafting projects through the lens of PBL. This finding gains support from the differentiation between disciplinary and meta-disciplinary aspects.

Thirdly, the distinction drawn between disciplinary and meta-disciplinary aspects offers valuable insights into how schools navigate the inherent complexities of STEAM PBL. This differentiation provides a framework for more focused changes in STEAM projects. Utilising both disciplinary and meta-disciplinary criteria holds potential implications for high school centres, which vary according to the extent to which the Science and Technology curriculum is delivered through STEAM PBL methodologies.

In this paper, we have seen that despite selecting a sample of secondary schools that are networked and share their thoughts and STEAM PBL instructional designs, there are still very particular trends on how to design STEAM projects that emerge when evaluating disciplinary and meta-disciplinary criteria.

This paper also raises concerns about how STEAM is pushing for the intended purposes of creating a call for more and more diverse people into this field. By looking at the findings of how STEAM is represented in Catalonia (Spain), we believe that more focus should be placed on fostering students’ motivations to STEAM through leveraging the identified joyful [1] and socially relevant aspects of STEAM PBL initiatives and that this programme considers a more representative engagement into scientific contents and practices [50]. We aim for this paper to have implications regarding how schools can clearly present their STEAM projects by raising a higher awareness in relation to their motives and how they are aligned at STEAM’s project level, at a subject level and at school level.

8. Limitations

While the strength of this study relies on the interpretation of how STEAM projects are associated with a progression level (category) and that descriptive statistics allowed for a graphical representation of this interpretation, these results cannot compel statistical differences significantly. Furthermore, the rubric instrument was developed in the frame of a science education perspective, which may lack a more holistic perspective that enables it to include more specific aspects of the integration of Mathematics and Arts.