A Tool to Assess Architectural Education from the Sustainable Development Perspective and the Students’ Viewpoint

Abstract

Architectural education plays a fundamental role in achieving sustainable development by training future professionals who can contribute to sustainability through their practice. Nevertheless, to ensure the introduction of sustainable development in the education of future architects, it is fundamental to understand what is being done and what could be improved. Despite this, a big gap has been detected in the assessment of architectural education in Spain and worldwide. Thus, a close-ended questionnaire for students has been designed based on the outcomes from a literature review, exploratory interviews with specialists, and the qualitative analysis of two schools of architecture. Additionally, it has been revised by experts, through the pilot study of a preliminary version, and with the supplementary analysis of the answers to a final test with students from different schools from Spain. In particular, this questionnaire allows for comprehensively measuring the students’ perception of their sustainability learning outcomes, their learning experience, and the connection between, with the aim of facilitating the adjustment of Spanish architectural education towards the introduction or enhancement of sustainable development by education managers, teachers, policymakers, and professional associations.

1. Introduction

Higher Education has been called to become Education for Sustainable Development (ESD) [1,2,3,4,5], a “holistic and transformational education which addresses learning content and outcomes, pedagogy and the learning environment” for “transforming society” by empowering “learners to take informed decisions and responsible actions for environmental integrity, economic viability and a just society, for present and future generations, while respecting cultural diversity” [6] (p. 12). Nevertheless, although teaching institutions have been integrating sustainability in education for years and good practices can be found in many different countries and at all levels and contexts of education [7,8,9,10,11,12,13], the much-needed reorientation towards sustainability has yet to take place [14,15,16,17,18,19,20,21,22]. To do so, it is fundamental to recognize what is being done and what could be improved [7,8].

With this aim, generalist Architectural Education (AE) has been assessed from the sustainability perspective in Spain and worldwide in relation to three main aspects: curricula—understood as a plan for formal learning—[23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52], the learning experience provided by schools [26,27,28,31,32,33,34,40,41,43,44,45,46,47,49,50,52,53,54,55], and the learning outcomes acquired by students [26,28,43,53,54,56,57,58,59,60,61,62,63].

To conduct these assessments, different methods involving different types of participants have been used:

• Curricula has mostly been assessed through document analysis [23,25,26,28,29,30,32,33,34,35,36,37,38,39,42,43,44,45,46,47,48,49,50,51,52];
• Learning experience has mainly been evaluated through interviews [26,32,33,34,43,44,45,46,47,52,55] or questionnaires [26,27,28,31,43,49,53,54], addressed to students [26,28,43,49,53,54], teachers or education managers [26,27,31,32,33,34,43,44,45,46,47,49,52,55];
• Learning outcomes have been primarily assessed with questionnaires for students [26,28,43,53,54,56,57,58,59,60,61,62,63].

According to key literature on ESD and Architectural Education for Sustainable Development (AESD), different factors must be analyzed for assessing these aspects:

• In the case of curricula: learning contents and their organization, learning methods, learning assessment, and learning outcomes [64,65];
• In the case of the learning experience: formal learning (understood as the curricula implementation: the learning contents addressed in each course, the learning methods used, the learning assessment conducted, and the learning outcomes promoted), non-formal learning (extra-curricular activities: courses or programs, internships, studying abroad, etc.), and informal learning supported by teachers and centers (operations and culture, extension, research, other education, and governance) [6,65,66,67,68,69,70,71];
• In the case of the learning outcomes acquired by students: the different levels of domain (according to different authors, these can be classified as awareness, knowledge, skills, behaviors or attitudes [56,57,58,72,73]; know, live together, do and be [74]; or cognitive, socio-emotional and behavioral [66,75]), the specific outcomes related to AESD and the broad practice of the generalist architect [5], and the cross-cutting competencies for sustainability [4,75];
• Furthermore, the three dimensions of sustainability—environmental, social, and economic—should be considered [6,67,75].

Nevertheless, a big gap has been detected concerning the factors addressed when assessing AE curricula and learning experience and outcomes from the perspective of sustainable development (SD):

• In the case of curricula, assessments on SD have focused on the introduction of environmental contents or outcomes and their organization in courses [23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52], covering only in few cases the learning methods and assessment planned [44,45,46,47,48,49,50,52] or the social and/or economic dimensions of SD [23,25,27,28,29,30,31,35,36,37,39,41,42,43,49,51] whilst overlooking in all instances the cross-cutting competencies for sustainability;
• In the case of the learning experience, evaluation on SD has focused on formal learning, covering the actual introduction of environmental contents and their organization in courses [26,27,28,31,32,33,34,40,41,43,44,45,46,47,49,50,52,53,54,55] and considering only in few cases the learning methods used [28,32,33,34,40,43,44,45,46,47,49,52,55], the students’ assessment conducted [28,32,33,34,40,44,45,46,47,52], and the presence of the social and/or economic dimensions of SD [27,28,31,43,49,53,55]. In all cases, the learning outcomes promoted by the learning process and informal learning have been ignored, and only in one study non-formal learning was addressed [55];
• In the case of learning outcomes for SD, existing evaluations have only covered the different levels of domain in one case [56,57,58] while overlooking some of them in the rest [26,28,43,53,54,59,60,61,62,63]. These studies have occasionally considered social and/or economic dimensions of sustainability [28,43,53,61] and have in all cases ignored the cross-cutting competencies for sustainability and architectural competencies unrelated to design or building design.

Thus, despite the preference for assessing curricula through documentary analysis and the learning experience and outcomes through questionnaires for students and/or teachers or education managers, there are no comprehensive and systematic tools or approaches in AE to jointly or separately evaluate curricula, learning experience, and learning outcomes from the SD perspective.

In this context, it is particularly critical to advance in the assessment of the learning experience and outcomes since this is where the main gap resides, and a better understanding of these two aspects would offer a complementary but fundamental perspective to improve AESD [4,6,66,67,75,76]. From a methodological point of view, this needed investigation should consider the proven effectivity that questionnaires can have in this kind of assessment and the key role that students can play as final recipients of the learning process [28,43,49,53,54,56,57,58,60]. Moreover, the process and result of the development of such a students’ questionnaire, would also provide a deeper understanding of the factors affecting AESD and would furnish education managers and pedagogical experts with a highly systematized and transferable tool, easily and economically applicable at different schools that allows for comparative studies between academic programs.

In this regard and on the contrary to architecture, other disciplines have comprehensively assessed the students’ learning experience and outcomes separately at Higher Education (HE) through questionnaires, with relevant works in Spain and worldwide [77,78,79,80,81,82,83,84,85,86,87,88,89,90,91]. Despite these questionnaires being designed for disciplines other than architecture or being addressed to teachers, they could provide valuable input for evaluating simultaneously the AESD learning experience and outcomes through the students’ perspective. In particular, two reference tools have been found especially comprehensive:

• Concerning the learning experience, the questionnaires Autodiagnóstico del Profesorado en Sostenibilización Curricular APROSOS [89,90,91] and Sostenibilidad y Práctica Docente [85,86,87,88] provide a comprehensive perspective of formal learning and informal learning supported by teachers, although they are both addressed to teachers and overlook non-formal education and informal education supported by centers;
• Regarding learning outcomes, the questionnaire from the EDINSOST Project [13,77,82,92,93,94,95] covers the holistic, environmental, social, and economic dimensions of SD, the cross-cutting competencies for sustainability, the overall professional practice of the degree to which it is adapted, and, in a simplified way, the four levels of domain defined by Miller [77,92,96]—know, know-how, demonstrate, and do. Moreover, this questionnaire is addressed to students and has been designed to be adjusted to different degrees.

In response to this situation and in line with the highlighted work, the authors of this paper investigate which contents, structure, and production process should a questionnaire for students have to provide a comprehensive and consistent understanding and assessment of AESD with a special focus on two critical aspects: learning experience and learning outcomes for SD. This research is conducted through the design and validation of a close-ended questionnaire for students from the Spanish qualifying Master’s Degree in Architecture (Máster Universitario en Arquitectura, in Spanish) and is aimed to facilitate the adjustment of AESD in Spain by education managers, teachers, policymakers, and professional associations.

The information obtained with this kind of questionnaire, together with curricula assessment, the perspective of other AE agents (education managers, pedagogical experts, teachers, practitioners, etc.), and additional approaches (documentary analysis, interviews, seminars, etc.), is considered essential to inform future adjustment of Spanish AE towards AESD. Only thus, it will be possible to ensure that future architects acquire the learning outcomes they need to practice an Architecture for Sustainable Development (ASD), which, according to the International Union of Architects, helps the United Nations Sustainable Development Goals to be achieved [97,98,99] by “[contributing] to the built environment and [making] choices that change the world for the better—through better buildings, settlements, landscape architecture, and urban planning” [97] (p. 1).

In particular, the conducted research and its final result (a questionnaire for assessing Spanish AESD through the students’ viewpoint) has been designed to answer the following research questions:

• How should a questionnaire be designed to evaluate students’ perception of their learning outcomes related to AESD?
• How should a questionnaire be designed to evaluate students’ perception of their learning experience regarding formal, non-formal, and informal AESD?
• How should a questionnaire be designed to evaluate students’ perception of the learning experience contribution to acquiring the aforementioned learning outcomes?
• How should a questionnaire be designed to evaluate the potential influence of students’ academic, professional, and personal background on the aforementioned aspects?

2. Materials and Methods

The process that was followed to answer the research questions and, subsequently, to design and validate the questionnaire presented in this work is shown in Figure 1. This process was planned based on key literature on research methodologies and surveys and questionnaires design [100,101,102,103,104,105,106,107,108]. Every step of the research was developed by the corresponding first author and supervised by coauthors in regular meetings and deliverables reviews. In particular, the research was conducted from November 2018 to June 2021 (see

Table 1. Timeline of the tasks performed to design the questionnaire.

Task	2018		2019												2020												2021
	11	12	1	2	3	4	5	6	7	8	9	10	11	12	1	2	3	4	5	6	7	8	9	10	11	12	1	2	3	4	5	6
A1 1	•
A2 1		•	•	•	•	•	•
A3 1		•	•	•	•	•	•
B 1		•	•	•	•	•	•	•
C 1		•	•	•	•	•	•	•	•
D 1											•	•	•	•	•
E 1																•	•
F 1																		•	•	•	•		•
G1 1																								•
G2 1																								•
G3 1																								•
H 1																								•	•
I1 1																									•	•	•	•	•	•	•	•
I2 1																																•
J 1											•	•	•	•	•	•	•	•	•	•	•	•	•	•	•	•	•	•	•	•	•	•

¹ See Figure 1.

).

The design of the questionnaire has been based, principally, on the contribution of three prior works on Higher Education for Sustainable Development (HESD) (see Section 1): the EDINSOST Project [13,77,82,92,93,94,95] and the questionnaires Autodiagnóstico del Profesorado en Sostenibilización Curricular APROSOS [89,90,91] and Sostenibilidad y Práctica Docente [85,86,87,88], adapted and completed for assessing AE and the students’ perspective with additional literature about ESD [6,65,67,68,69,70,75,109,110,111,112], AESD [24,32,64,71,76,113,114,115], and ASD [97,98,99,116,117,118,119,120,121,122].

2.1. Grounding, Conceptualization, and Operationalization

A literature review on questionnaires design (see task A1 in Figure 1), exploratory interviews with 18 experts (see task A2 in Figure 1), and the qualitative analysis of two schools (see task A3 in Figure 1) were conducted to substantiate the design of the questionnaire. Afterward, based on the outcomes from these grounding tasks and the main references selected, the questionnaire variables and values were generated through conceptualization and operationalization processes (see tasks B and C in Figure 1).

2.1.1. Grounding

Literature about surveys and questionnaires design [100,101,102,103,104,105,106,107,108] was reviewed to complement the literature on ESD, AESD, and ASD (see task A1 in Figure 1).

In addition, exploratory interviews and conversations with 18 experts in SD, ESD, ASD, AESD, and HE assessment were conducted (see task A2 in Figure 1). Among them, there were 4 experts in SD and 6 experts in ESD from Europe, 4 experts in ASD from Spain, 4 experts in AESD from Europe, and 2 experts in HE assessment from Spain. Moreover, at least six of them had previous experience in questionnaires design. In particular, these experts participated in the initial grounding of the questionnaire and some of them also during the design process.

Furthermore, the qualitative analysis of two schools was performed (see task A3 in Figure 1): the School of Architecture of the Universitat Politècnica de València (UPV, Spain) and the School of Arts, Design and Architecture of Aalto University (AaU, Finland), as an exemplary one in the field. The analysis at the UPV was performed from December 2018 to February 2019, and at the AaU from March to May 2019. These qualitative analyses were conducted through interviews with students’ representatives, teachers, and managers, documentary analyses, and the observation of courses and final juries.

2.1.2. Conceptualization and Operationalization

Firstly, the constructs, dimensions, subdimensions, and indicators—or the concepts and properties—of the questionnaire were established based on the outcomes from the grounding tasks and on the main references selected (see task B in Figure 1 and Figure 2).

Afterward, these properties or indicators were operationalized for generating the questionnaire variables and values (see task C in Figure 1 and Figure 2).

2.2. Design, Revision, and Adjustment

After designing the first draft of the questionnaire (see task D in Figure 1), a revision-adjustment process was performed in two steps. First, the initial draft was reviewed by 16 experts and adjusted by the corresponding authors to develop the second draft (see task F in Figure 1). Secondly, a pilot test of the second draft was conducted with 17 students from the UPV (see task G1 in Figure 1), with a statistical analysis of the results (see task G2 in Figure 1) and a focus group with participants (see task G3 in Figure 1), providing qualitative and quantitative feedback that allowed to adjust this draft and to design the final version of the questionnaire (see task H in Figure 1).

2.2.1. Design and Adaptation of Measurements, Scales, Questions, and Responses

To design the first draft, the kind of measurements, scales, questions, and responses were decided and the questionnaire items (questions) were written or adapted from references (see task D in Figure 1).

2.2.2. Revision of the First Draft by 16 Experts

The first draft of the questionnaire was revised by 16 experts between February and March 2020 (see task E in Figure 1). Given the transdisciplinary nature of the research, some of them were experts in ESD, others in ASD, and the rest in AESD. They were selected based on their background in the field and on the condition that they spoke Spanish, as it is the language in which the questionnaire was designed. First, 5 Spanish experts in ESD assessed the whole questionnaire without focusing on ASD-related issues. Secondly, 4 experts in ASD assessed the architecture-related questions, focusing only on disciplinary issues. Finally, 7 Spanish and international experts in AESD assessed the whole questionnaire from a comprehensive perspective. Among them, at least 9 had experience in questionnaires design, 3 in connection with AESD and 6 concerning ESD.

The experts’ consultation was designed based on the Delphi method [123,124], with iterative consultation rounds aimed to reach consensus and stability in answers through feedback. To do so, the participants were provided with a form to assess the questionnaire. In particular, this form allowed to quantitatively and qualitatively assess each part of the questionnaire and to provide an overall evaluation. Moreover, it included a section to qualitatively evaluate each question or text of the questionnaire, so that experts could indicate which ones should be removed, added, or improved. Furthermore, they were asked to include additional comments or general considerations about the questionnaire and its different parts. This form was created based on existing works [125,126].

Due to the coronavirus disease 2019 (COVID-19) crisis, which started at the same time as the revision of the questionnaire, 5 experts completed the form. The rest of the experts provided qualitative feedback by emailing free texts with their perspectives about the design of the questionnaire or by arranging a meeting to discuss it.

Despite this situation, feedback was found enriching and aligned, and a new assessment form was prepared with the conclusions of the first round to confirm the stability of opinions and to increase the consensus regarding any conflicting point of view. Four answers were received in this case. Nevertheless, the consensus and the stability in the opinions received through the form and in those obtained by email or in person was high and allowed for adjusting the first draft. In particular, apart from some notes about specific items, the experts principally suggested shortening the questionnaire and reformulating the dimensions and questions related to the “AESD Learning Outcomes” construct, including some suggestions to address this issue.

2.2.3. Adjustment of the First Draft

Based on the outcomes from the experts’ revision, a second draft was developed and hosted in the online surveying platform used in the final version: LimeSurvey Community Edition (LimeSurvey GmbH, Hamburg, Germany), version 3.23.3+200909, downloaded, installed, and maintained at the UPV servers (see task F in Figure 1).

2.2.4. Pilot Test of the Second Draft with 17 Students, Statistical Analysis, and Focus Group with Participants

Once the second draft of the questionnaire was finished in October 2020, a pilot test was conducted with a sample of 17 students from the 5th year of the Bachelor’s Degree in Architecture (Grado en Fundamentos de la Arquitectura, in Spanish) from the UPV (see task G1 in Figure 1) to obtain useful feedback without altering the results from the subsequent test of the final version with the Master’s students (see Section 2.3). A non-random sampling was done without prior notice by presenting the questionnaire in school time to 22 students attending one course on a certain day and providing them an individual password to access the questionnaire. They were asked to voluntarily complete it online, outside school time, during the following week.

Despite existing recommendations to use bigger sample sizes in pilot tests [127,128]; it was not possible to perform it in better conditions. To compensate for this situation, and to delve into the analysis of the questionnaire, a statistical analysis (see task G2 in Figure 1) and a focus group with participants (see task G3 in Figure 1) [107,129] were performed after collecting the students’ answers.

Firstly, the statistical analysis (see task G2 in Figure 1) was conducted to evaluate each item individually and the overall questionnaire (n = 17). On the one hand, to assess the items individually, the proportion of chosen answers was analyzed. On the other hand, to assess the reliability of the overall questionnaire, the internal consistency of the scales was measured through the Cronbach’s alfa indicator [130,131,132,133,134,135,136,137], in line with the reliability assessment conducted with the questionnaires used as a reference [19,77]. Additionally, the proportion of “Don’t know” answers were assessed [101]. Finally, the length of the questionnaire and each of its parts was studied. In particular, this data was obtained by using the statistical software SPSS Statistics (IBM Corp, Armonk, NY, USA), version 26, for Macintosh OS, and the statistical package included in the surveying platform Limesurvey (see Section 2.2.3), although other software may also be suitable (R-Studio—R-Studio, Boston, MA, USA; Statistica—StatSoft, Hamburg, Germany; Stata—StataCorp LLC, College Station, TX, USA; GNU Regression, Econometrics and Time-series Library—Cottrell, A., Winston-Salem, NC, USA; and Lucchetti, R.J., Ancona, Marche, Italy—; etc.).

Secondly, the day after the deadline to answer the second draft of the questionnaire, a focus group discussion was conducted during school time with the respondents (see task G3 in Figure 1) to get feedback about the questionnaire’s completion. In particular, different issues arising from the statistical analysis mentioned above were debated together with other topics regarding the length and the structure of the questionnaire, the relevance of the information, and the clarity of the questions, the response options, and the information included.

At the statistical analysis, no issues were found concerning the internal consistency of the scales and the proportion of “Don’t know” answers. Nevertheless, some response patterns did not respond to the logic behind the hierarchy of the levels of domain, as it was expected to find higher levels of competence in lower levels of domain than in higher ones. In particular, the questions behind these results and other ones were identified as confusing by participants during the focus group discussions and were slightly adapted. Additionally, the questions with a higher proportion of “Don’t know” answers were analyzed, concluding that the preference for this response option derived from the students’ lack of awareness about the issue asked. Furthermore, in the focus group discussion, students concluded that the questionnaire length was excessive, thus, it was resolved to distribute it during school time and some dimensions were simplified. In particular, all these issues were easily solved.

2.2.5. Adjustment of the Second Draft

Based on the received feedback, the final version of the questionnaire was developed from October 2020 and finished in November 2020 (see task H in Figure 1).

2.3. Test and Quality Analysis of the Final Version of the Questionnaire

Once the final version of the questionnaire was ready, it was distributed in 19 Spanish Schools of Architecture to evaluate its quality through a descriptive study and a reliability assessment (see tasks I1 and I2 in Figure 1). In particular, 261 students (n) from the qualifying Master’s Degree in Architecture completed the final questionnaire online (see Appendix A). To access the questionnaire, individual passwords were distributed from November 2020 to June 2021 at each school of architecture together with a presentation of the research. Afterward, the questionnaire was self-administered by the students in different ways (see Appendix A) and reminders were sent to trigger participation.

Again, the descriptive study of the students’ answers during this final test, was aimed to assess the overall questionnaire and each item individually. Firstly, to assess the questionnaire from a general perspective, the proportion of “Don’t know” answers were evaluated. To do so, the frequency of selection of this answer was divided by the total number of responses [101,106]. On the other hand, the questionnaire length was calculated in one of the schools (Universidad Rey Juan Carlos), in which the questionnaire was completed under supervision during school time (see Appendix A). Moreover, missing data was evaluated by calculating the total number of students that started but did not complete the questionnaire. Secondly, to assess each item or question individually, the proportion of selected alternatives of the close-ended responses was studied. Despite not easily predicting in advance the “appropriate” response pattern [101], it was observed if some alternatives were always or never selected, if the answer “Don’t know” was marked above average, and if the learning outcomes answers followed a logic regarding the different levels of domain, considering that the upper levels of domain should show lower levels of competency.

On the other hand, the reliability of the final version of the questionnaire was assessed by analyzing the internal consistency of its scales through the Cronbach’s alpha coefficient, according to different literature [130,131,132,133,134,135,136,137] and to the reliability assessment performed with the questionnaires used as a reference [19,77]. This measure indicates the extent to which the different items of a scale are closely related as a group. In particular, values of 0.7 or higher are considered acceptable [130,131,132,133], although some authors also refer to acceptable values of 0.6 or higher [62,138,139,140].

In particular, the prior analyses were conducted by using the same software as in the pilot test (see Section 2.2.4).

2.4. Transversal Tasks and Ethics

In parallel to the prior tasks, many transversal issues were considered to inform the definition of the questionnaire, such as available resources, limitations of the research, format, and administration of the questionnaire, etc. Simultaneously to the design, the revision, and the adjustment of the questionnaire, the plan for its administration to students and its formatting was done. Furthermore, the privacy policy, the informed consent, and the research ethical conditions were designed by the research team and revised and approved by the Ethics Committee of the UPV (Project code: P3-07-05-2020) (see task J in Figure 1).

3. Results

The questionnaire detailed below is the result of the research presented in this work. An English translation of the validated questionnaire (in Spanish) can be found in Supplementary Materials as Document S1.

3.1. Aim and Objective of the Questionnaire

The questionnaire aims to facilitate a higher alignment between Spanish AE and SD. To do so, the objective of the questionnaire is to assess the Master’s degree students’ perspective towards:

• their learning outcomes, in the field of architecture and sustainable development (Construct 1: AESD Learning Outcomes);
• their learning experience, regarding the qualities of AESD (Construct 2: AESD Learning Experience)
• the extent to which their learning experience has contributed to acquiring the aforementioned outcomes (Construct 3: “AESD Learning Experience” contribution to “AESD Learning Outcomes” acquisition);
• some of their academic, professional, and personal background that may influence their answers and be necessary for sampling and for assessing the sample representativeness.

Moreover, the questionnaire intends to allow the correlational study of the results from the aforementioned constructs and their dimensions, subdimensions, indicators, and variables.

3.2. Scope of the Questionnaire

The questionnaire has been designed to facilitate an exploratory-descriptive and correlational study. It can be used to perform longitudinal and cross-sectional studies of single and/or multiple cases.

3.3. Population of the Questionnaire

The questionnaire is addressed to students from the Spanish qualifying Master’s Degree in Architecture (Máster Universitario en Arquitectura, in Spanish) [141]. It is a one-year (60 ECTS credits) degree that qualifies for accessing the regulated profession of architecture in Spain, which can only be studied after graduating from the five-year (300 ECTS credits) Bachelor’s Degree in Architecture (Grado en Fundamentos de la Arquitectura or Grado en Estudios de Arquitectura, in Spanish). This academic program is recognized by the European professional qualifications Directive 2005/36/EC [142].

3.4. Administration of the Questionnaire

The questionnaire has been designed to be presented and self-administered in the classroom, during school time. It must be completed at the online platform LimeSurvey (see Section 2.2.3) on the students’ or school electronic devices, ideally, on computers. The questionnaire length is detailed in Section 3.8.3. To complete the questionnaire, it is mandatory to answer all the items.

3.5. Structure of the Questionnaire

The questionnaire has been structured in seven parts:

• Presentation (A);
• Privacy policy and informed consent (B);
• Questions about your academic background (C);
• Questions about your AESD learning outcomes (D);
• Questions about your AESD learning experience and its contribution to acquiring the AESD learning outcomes (E);
• Questions about your academic, professional, and personal background (F);
• Thank-you message (G).

3.5.1. Presentation

This part includes the presentation of the questionnaire. It explains the importance of participating in the survey, the details of the research team behind, the population to which it is addressed, the aim and objectives of the questionnaire, the types of questions that will be asked, the voluntary and anonymous nature of the participation, and a thank-you note.

3.5.2. Privacy Policy and Informed Consent

This part includes the privacy policy and the informed consent of the questionnaire. It explains the conditions of participation and asks the participants for their explicit and informed acceptance as a condition to enter the questionnaire.

3.5.3. Questions about Your Academic Background

This part includes 5 close and open-ended items (C.1–C.5) related to the students’ “Academic information” (C.1–C.5) and their “SD and/or ASD training compared to classmates” (C.4 and C.5) (see Section 3.7).

3.5.4. Questions about Your AESD Learning Outcomes

This part includes 60 close-ended items, grouped in 10 sets (D.1–D.10), related to the students’ “AESD Learning Outcomes” (Construct 1, see Section 3.6).

3.5.5. Questions about Your AESD Learning Experience and Its Contribution to Acquiring the AESD Learning Outcomes

This part includes 104 close-ended items, grouped in 11 sets (E.1–E.11). They mostly relate to the students’ “AESD Learning Experience” (Construct 2) (E.1–E.9) and to their perspective on the “AESD Learning Experience contribution to AESD Learning Outcomes acquisition” (Construct 3) (E.10) (see Section 3.6). Furthermore, one item addresses the students’ “Satisfaction with the degree” (E.11) (see Section 3.7).

3.5.6. Questions about Your Academic, Professional, and Personal Background

This part includes 25 close and open-ended items, grouped in 22 sets (F.1–F.22). They relate to the students’ “Involvement in the degree” (F.4–F.5), “SD and/or ASD training compared to classmates” (F.6–F.8, F.10 and F.14–F.19), “Academic self-concept” (F.2 and F.3), “Academic information” (F.1, F.9 and F.11–F.13), and “Personal information” (F.20–F.22) (see Section 3.7).

3.5.7. Thank-You Message

This part includes a thank-you message for the participants in the questionnaire.

3.6. Constructs, Dimensions, Indicators, Items, Variables, and Values of the Questionnaire

The questionnaire addresses the students’ perception of the following constructs related to AESD (see Figure 3):

• AESD Learning Outcomes (Construct 1);
• AESD Learning Experience (Construct 2);
• “AESD Learning Experience” contribution to “AESD Learning Outcomes” acquisition (Construct 3).

The questionnaire constructs, dimensions, subdimensions, indicators, items, and variables are detailed below. In particular, the items from each construct, dimension, and/or subdimension constitute a scale (see Figure 2).

3.6.1. AESD Learning Outcomes

This construct (Part D of the questionnaire) includes the dimensions, subdimensions, indicators, and items displayed in

Table 2. Dimensions, subdimensions, indicators, and items of the construct “AESD Learning Outcomes”, from the perspective of the four “cross-curricular skills for sustainability” from CRUE.

CONSTRUCT 1: AESD Learning Outcomes (0.958/0.949–0.967) 1
Dimensions	Subdimensions	Indicators	Items
C1. Competence in the critical contextualization of knowledge through the linking of social, economic, and environmental issues on a local and/or global level (0.660/0.585–0.708) 1	Holistic	L1. Knows	D.1.a, D.2.a, D.2.b
		L2. Knows how	D.2.c
		L3. Demonstrates	D.2.d
		L4. Does	D.9.a
		L5. Is	D.10.a
C2. Competence in the sustainable use of resources and in the prevention of negative impacts on natural and social environments (0.946/0.944–0.949) 1	Environmental (0.795/0.740–0.854) 1	L1. Knows	D.3.1.a, D.3.2.a
		L2. Knows how	D.3.1.b, D.3.2.b
		L3. Demonstrates	D.3.1.c, D.3.2.c
		L4. Does	D.9.b
		L5. Is	D.10.c
	Social (0.817/0.777–0.839) 1	L1. Knows	D.4.1.a, D.4.2.a
		L2. Knows how	D.4.1.b, D.4.2.b
		L3. Demonstrates	D.4.1.c, D.4.2.c
		L4. Does	D.9.c
		L5. Is	D.10.d
	Economic (0.855/0.810–0.906) 1	L1. Knows	D.5.1.a, D.5.2.a
		L2. Knows how	D.5.1.b, D.5.2.b
		L3. Demonstrates	D.5.1.c, D.5.2.c
		L4. Does	D.9.d
		L5. Is	D.10.e
	Holistic (0.880/0.868–0.889) 1	L1. Knows	D.1.b, D.1.c, D.1.d, D.1.e, D.6.1.a, D.6.2.a
		L2. Knows how	D.6.1.b, D.6.1.c, D.6.1.d, D6.2.b
		L3. Demonstrates	D.6.1.e, D.6.1.f, D.6.2.c
		L4. Does	D.9.e
		L5. Is	D.10.b
C3. Competence to participate in community processes that promote sustainability (0.809/0.765–0.847) 1	Holistic	L1. Knows	D.7.a, D.7.b, D.7.c
		L2. Knows how	D.7.d
		L3. Demonstrates	D.7.e, D.7.f
		L4. Does	D.9.f
		L5. Is	D.10.f
C4. Competence to apply ethical principles related to sustainability values in personal and professional behavior (0.740/0.646–0.788) 1	Holistic	L1. Knows	D.8.a, D.8.b
		L2. Knows how	D.8.c
		L3. Demonstrates	D.8.d
		L4. Does	D.9.g
		L5. Is	D.10.g

¹ Results of the analysis of the internal consistency of the scales that constitute the construct and each dimension and subdimension (Scale Cronbach’s alfa/Scale alfa range if an item is removed, see Section 2.3).

and will be used to define the perception of the students towards their learning outcomes concerning architecture and sustainable development.

The dimensions, subdimensions, indicators, and items from Table 2, and the variables and values behind them, have been defined based on the work from the EDINSOST project [13,77,82,92,93,94,95] (see Section 2).

In particular, each item includes a question about a particular “learning outcome” from the sustainability competency map in Appendix B, which has been adapted to generalist architecture degrees based on the EDINSOST sustainability map for engineering degrees [92].

First, each “learning outcome” corresponds to one of the four “cross-curricular skills for sustainability” defined by the Association of Spanish University Rectors (CRUE, by its Spanish acronym) for inclusion in university education [4]. These competencies constitute the four dimensions behind the construct “AESD Learning Outcomes”: C1. Competence in the critical contextualization of knowledge through the linking of social, economic, and environmental issues on a local and/or global level; C2. Competence in the sustainable use of resources and the prevention of negative impacts on natural and social environments; C3. Competence to participate in community processes that promote sustainability; and C4. Competence to apply ethical principles related to sustainability values in personal and professional behavior. As displayed in Figure 3, each dimension is studied holistically (H) except for C2, which is also analyzed from the environmental (EV), the social (S), and the economic (EC) perspectives as subdimensions, as it does the EDINSOST project [92]. Additionally, the dimension C2, and its subdimensions (H, EV, S, and EC), can be divided into two new subdimensions and scales each, with some “learning outcomes” (or items) connected with the “sustainable use of resources and the prevention of negative impacts in architecture-related projects and actions”, and others related to “the use of tools and metrics to estimate these” (see Appendix C).

Additionally, each “learning outcome” corresponds to one “level of domain” from Miller’s pyramid taxonomy [96]: L1. Knows, L2. Knows how, L3. Demonstrates, and L4. Does; completed with an attitudinal level: L5. Is, based on the work from Delors et al. [74], the AKASA model [56,57,58,72,73], Pooley et al. [143], and Ajzen’s Theory of Planned Behavior [144,145]. Thus, L1 refers to knowledge, L2 relates to the integration of knowledge and abilities, L3 concerns the ability to perform the competency in action, L4 bears on the actual behavior [96,146], and L5 is connected with attitudes and values. According to this perspective, the construct “AESD Learning Outcomes” can also be interpreted towards these levels, considering each one as a different dimension and scale (see Appendix D).

As it can be seen in Document S1, to answer the questions related to the levels of domain L1–L4 (items D.1–D.9), the students’ have to state the extent to which they agree that the statement shown towards their degree of acquisition of a particular “learning outcome” represents themselves, with a 4-point Likert scale with the following variables and values: “Strongly disagree” (0), “Disagree” (1), “Agree” (2), “Strongly agree” (3), “Don’t know” (4). On the other hand, to answer the questions regarding the level of domain L5 (items D.10.), the students have to state the extent to which they feel favorable versus unfavorable towards the statement shown—which represents an attitude object [143,147]— with a 4-point Likert scale with the following variables: “Extremely unfavorable”, “Unfavorable”, “Favorable”, “Extremely favorable”, “Don’t know”. According to the positivity or negativity of the attitude object underlying the statement, the values of the variables are 0, 1, 2, 3, and 4, or 3, 2, 1, and 4, respectively (see Document S1).

3.6.2. AESD Learning Experience

This construct (Part E of the questionnaire) includes the dimensions, subdimensions, indicators and items displayed in

Table 3. Dimensions, subdimensions, indicators, and items of the construct “AESD Learning Experience”.

CONSTRUCT 2: AESD Learning Experience (0.983/0.983–0.983) 1
Dimensions	Subdimensions	Indicators						Items
Formal Education (0.980/0.979–0.980) 1	Methods (0.877/0.864–0.877) 1	Intradisciplinary transversality						E.1.a
		Transdisciplinarity						E.1.b
		Service, reality, and context-based learning						E.1.c
		Local-global perspective						E.1.d
		Short, medium, and long-term perspectives						E.1.e
		Pedagogical strategies and techniques to facilitate expression and contrast ideas and reflection or action on topics						E.1.f
		Active participation, interaction, and cooperative learning						E.1.g
		Debates						E.1.h
		Individual, pair, and group work						E.1.i
		Lectures and masterclasses						E.1.j
		Digital learning resources						E.1.k
		Resources to support and/or deepen learning						E.1.l
		Students’ involvement in the design of the teaching-learning process						E.1.m
		Combination of the previous situations in courses						E.1.n
	Assessment (0.919/0.912–0.918) 1	Students’ awareness of the assessment criteria and forms						E.2.a
		Students’ involvement in the design of the assessment criteria and forms						E.2.b
		Clarity of the assessment criteria						E.2.c
		Assessment of the outcomes addressed						E.2.d
		Assessment objectivity						E.2.e
		Assessment comprehensiveness						E.2.f
		Assessment continuity						E.2.g
		Quantitative and qualitative assessment criteria						E.2.h
		Feedback immediacy						E.2.i
		Formative assessment						E.2.j
		Diversity of assessment tools						E.2.k
		Communicative hetero-evaluation, co-evaluation, and self-evaluation						E.2.l
		Intradisciplinary transversal perspective in assessment						E.2.m
		Interdisciplinary perspective in assessment						E.2.n
		Reality and context-related perspectives in assessment						E.2.o
		Local-global perspective in assessment						E.2.p
		Combination of the previous situations in courses						E.2.q
	Outcomes (0.952/0.948–0.951) 1	CRUE “cross-curricular skills for sustainability”		C1 2	Holistic		Addressed	E.3.a.1
							Evaluated	E.3.a.2
				C2 2	Environmental		Addressed	E.3.b.1
							Evaluated	E.3.b.2
					Social		Addressed	E.3.c.1
							Evaluated	E.3.c.2
					Economic		Addressed	E.3.d.1
							Evaluated	E.3.d.2
					Holistic		Addressed	E.3.e.1
							Evaluated	E.3.e.2
				C3 2	Holistic		Addressed	E.3.f.1
							Evaluated	E.3.f.2
				C4 2	Holistic		Addressed	E.3.g.1
							Evaluated	E.3.g.2
		Levels of domain		L1–L3 3	Knowledge and abilities		Addressed	E.4.a.1
							Evaluated	E.4.a.2
				L5 3	Attitudes		Addressed	E.4.b.1
							Evaluated	E.4.b.2
				L4 3	Behaviors		Addressed	E.4.c.1
							Evaluated	E.4.c.2
	Contents (0.956/0.952–0.959) 1	Spaces of contents introduction	Teachers’ discourse					E.5.a
			Courses objectives					E.5.b
			Courses competencies or learning outcomes					E.5.c
			Courses assessment criteria					E.5.d
			Courses materials and resources					E.5.e
			Classroom activities					E.5.f
			Coursework criteria					E.5.g
			Coursework topics					E.5.h
			Bachelor’s thesis criteria					E.5.i
			Bachelor’s thesis topics					E.5.j
		Types of contents regarding SD, ASD, and CRUE “cross-curricular skills for sustainability”	SD					E.6.a
			Architecture and SD			Holistic		E.6.b
						Environmental		E.6.c
Social						E.6.d
Economic						E.6.e
Decision making with SD criteria					E.6.f
Techniques to reflect on, and to consider, SD problems					E.6.g
Ethical implications of topics and professional responsibility					E.6.h
Global thinking, local-global interconnections, and context influence					E.6.i
Inter and transdisciplinary perspective and work					E.6.j
Types of contents regarding the levels of domain	Concepts, data, facts, or principles					E.7.a
	Rules, techniques, methods, abilities, strategies, or procedures					E.7.b
	Attitudes, values, or norms					E.7.c
Global perspective and Organization (0.860/0.836–0.862) 1	Level of introduction of SD per group of mandatory courses		Design studio				E.9.1.a
			Urbanism				E.9.1.b
			Architectural History, Theory, and Composition				E.9.1.c
			Construction				E.9.1.d
			Structures				E.9.1.e
			Systems				E.9.1.f
			Basic Sciences				E.9.1.g
			Drawing				E.9.1.h
			Bachelor’s thesis				E.9.1.i
	Level of introduction of SD per type of courses		Mandatory courses				E.9.2.a
			Elective courses				E.9.2.b
	Overall level of introduction of SD		Bachelor’s degree				E.9.3.a
Non-formal and Informal education (0.922/0.912–0.922) 1	Informal education supported by teachers (0.835/0.800–0.827) 1	Saving of energy						E.8.1.a
		Use of digital platforms						E.8.1.b
		Saving of paper						E.8.1.c
		Saving of other materials						E.8.1.d
		Minimization of risks for people or the environment						E.8.1.e
		Protection of diversity and equal treatment						E.8.1.f
		Support for students with special needs						E.8.1.g
		Emphasis on good practices towards SD						E.8.1.h
	Non-formal and informal education supported by the center (0.907/0.882–0.902) 1	Environmental responsibility and promotion of sustainable lifestyles						E.8.2.a
		Social responsibility and promotion of safety, health, and social justice						E.8.2.b
		Exemplification and promotion of commitment to sustainable development						E.8.2.c
		University extension						E.8.2.d
		Research on SD						E.8.2.e
		Education on SD						E.8.2.f
		Management and assessment of institutional sustainability						E.8.2.g

¹ Results of the analysis of the internal consistency of the scales that constitute the construct and each dimension and subdimension (Scale Cronbach’s alfa/Scale alfa range if an item is removed, see Section 2.3). ² C1. Competence in the critical contextualization of knowledge through the linking of social, economic, and environmental issues on a local and/or global level; C2. Competence in the sustainable use of resources and in the prevention of negative impacts on natural and social environments; C3. Competence to participate in community processes that promote sustainability; and C4. Competence to apply ethical principles related to sustainability values in personal and professional behavior. ³ L1. Knows, L2. Knows how, L3. Demonstrates, L4. Does, and L5. Is.

and will be used to define the students’ perception of their learning experience concerning the qualities of AESD.

The construct dimensions, subdimensions, indicators, items, variables, and values have been designed based on the following questionnaires, which were completed for adjusting them to AE and the students’ understanding and for addressing informal and non-formal learning supported by centers (see Section 2):

• Autodiagnóstico del Profesorado en Sostenibilización Curricular APROSOS (Teachers’ Self-assessment in Curriculum Sustainability, in English), from the Grupo de Trabajo de Sostenibilización Curricular (Curriculum Sustainability Work Group, in English) of the CRUE [89,90,91];
• Sostenibilidad y Práctica Docente (Sustainability and Teaching Practice Questionnaire, in English) from Murga-Menoyo et al. [85,86,87,88].

In particular, each item includes a statement regarding the learning experience lived by the students towards a particular quality of AESD. To answer these items, the participants have to state the extent to which they feel that their learning experience corresponded to the statement shown with a 4-point Likert scale with the following variables and values: “Not at all” (0), “A little” (1), “Some” (2), “A lot” (3), “Don’t know” (4).

Due to the structure of the indicators, the following subdimensions could be divided into new subdimensions and scales: “Competencies” (see Appendix E), “Contents” (see Appendix F), and “Global perspective and Organization” (see Appendix G), so that they can be studied with further detail.

3.6.3. “AESD Learning Experience” Contribution to “AESD Learning Outcomes” Acquisition

This construct (Part E of the questionnaire) includes the dimension, indicators, and items displayed in

Table 4. Indicators and items of the construct “AESD Learning Experience contribution to AESD Learning Outcomes acquisition”.

CONSTRUCT 3: “AESD Learning Experience” Contribution to “AESD Learning Outcomes” Acquisition (0.887/0.822–0.863) 1
Indicators	Items
L1–L3. Knowledge and abilities	E.10.a
L5. Attitudes	E.10.b
L4. Behaviors	E.10.c

¹ Results of the analysis of the internal consistency of the scale that constitutes the construct (Scale Cronbach’s alfa/Scale alfa range if an item is removed, see Section 2.3).

and will be used to define the students’ perception of the extent to which their “AESD Learning Experience” has contributed to developing their “AESD Learning Outcomes”.

Each item addresses a different level of domain from the “AESD Learning Outcomes”: “Knowledge and abilities” (referring to L1. Knows, L2. Knows how and L3. Demonstrates), “Behaviors” (concerning L4. Does), and “Attitudes” (regarding L5. Is). To answer these questions, the students have to state the extent to which they feel that their learning experience has contributed to acquiring, from an overall perspective, the different “AESD Learning Outcomes” (see Table 2) at each level of domain (L1–L3, L4, and L5), with a 4-point Likert scale with the following variables and values: “Not at all” (0), “A little” (1), “Some” (2), “A lot” (3), “Don’t know” (4).

3.7. Properties, Items, Variables, and Values of the Questionnaire

Apart from the AESD-related constructs mentioned above (see Section 2.2.4, the questionnaire includes questions about the students’ academic, professional, and personal backgrounds that may influence their answers to Parts C and F of the questionnaire. These properties will be used to interpret data arising from the questionnaire constructs, and are grouped in the clusters shown in Figure 4. Moreover, data from the cluster “Academic information” will be used to conduct sampling, and data from the cluster “Personal information” will allow for assessing the sample representativeness.

The items related to the aforementioned clusters are displayed in

Table 5. Items concerning the students’ academic, professional, and personal background.

Concepts	Items
Satisfaction with the degree	E.11
Involvement in the degree	F.4, F.5.a, F.5.b
SD and/or ASD training compared to classmates	C.4, C.5, F.6, F.7, F.8, F.10, F.14, F.15, F.16, F.17, F.18, F.19.a, F.19.b, F.19.c
Academic self-concept 1	F.2, F.3
Academic information	C.1, C.2, C.3, C.4, C.5, F.1, F.9, F.11, F.12, F.13
Personal information	F.20, F.21, F.22

¹ To define the students’ academic self-concept, the information obtained in F.2. and F.3. will be compared with the grade point average of the student’s cohort.

. Although some items may be grouped and connected to the same idea, it is important to state that they have not been designed as a scale. These items have to be answered according to the variables and values detailed in Document S1.

3.8. Quality of the Questionnaire

To show the quality of the questionnaire, the answers to the final version from 261 students (n) from 19 Spanish Schools of Architecture were evaluated through a descriptive study and a reliability assessment (see Section 2.3). Below can be found the results of the following analyses:

• The overall proportion of “Don’t know” answers;
• The proportion of chosen answers at each question;
• Questionnaire length;
• Missing data;
• Internal consistency of the scales.

3.8.1. The Overall Proportion of “Don’t Know” Answers

The results of this study show that, despite the questionnaire length, the answer “Don’t know” was rarely selected in all the constructs by the 261 participants (see

Table 6. Percentage of selection of the response “Don’t know” at each construct (n = 261).

CONSTRUCT 1: AESD Learning Outcomes	CONSTRUCT 2: AESD Learning Experience	CONSTRUCT 3: “AESD Learning Experience” Contribution to “AESD Learning Outcomes” Acquisition
2.26%	4.02%	1.53%

). Additionally, the analysis showed that the proportion of selection of “Don’t know” was not higher in the items at the end of each part of the questionnaire.

3.8.2. The Proportion of Chosen Answers at Each Question

The outcomes from this analysis (n = 261) did not show any latent problems, although questions E.1.n, E.2.q, E.5.j, E.8.1.g, E.8.2.e, E.8.2.f, E.8.2.g and E.9.1.g presented a proportion of “Don’t know” answers significantly above average (10.34%, 11.88%, 16.09%, 10.73%, 15.33%, 22.61%, 27.97% and 16.86%, respectively). In particular, these results have been considered acceptable because they may not be caused by how the question was presented but from the unfamiliarity of the respondents to the asked issue, as it was already noticed in the pilot test of the second draft for these same questions (see Section 2.2.4).

3.8.3. Questionnaire Length

The outcomes from this analysis indicated that the average questionnaire length of the questionnaire is 30 min, with some students needing up to 40 min.

3.8.4. Missing Data

The results of this study show that from among the 324 students that accessed the questionnaire, only 284 answered the first 5 items (part C of the questionnaire). Then, only 264 completed the questions regarding the construct “AESD Learning Outcomes” (part D of the questionnaire). Finally, among these students only 261 completed the whole survey. In particular, the 20 students that stopped between parts C and D were completing the questionnaire on their own (see Appendix A). On the other hand, as answering all the questions was mandatory to submit the questionnaire, no missing data has been reported regarding the answers from the 261 students that completed it entirely.

3.8.5. Internal Consistency of the Scales

The outcomes from the internal consistency assessment (n = 261) can be found in the tables of each construct (scale Cronbach’s alfa/scale alfa range if an item is removed). Despite the good results obtained, the scales corresponding to the dimensions “C1. Competence in the critical contextualization of knowledge through the linking of social, economic and environmental issues on a local and/or global level” (see Table 2) and “L5. Is” (see Appendix D,

Table A4. Dimensions, subdimensions, indicators, and items of the construct “AESD Learning Outcomes”, from the perspective of the five levels of domain proposed.

CONSTRUCT 1: AESD Learning Outcomes (0.958/0.949–0.967) 1
Dimensions	Indicators		Items
L1. Knows (0.899/0.891–0.897) 1	C1 2	Holistic	D.1.a, D.2.a, D.2.b
	C2 2	Environmental	D.3.1.a, D.3.2.a
		Social	D.4.1.a, D.4.2.a
		Economic	D.5.1.a, D.5.2.a
		Holistic	D.1.b, D.1.c, D.1.d, D.1.e, D.6.1.a, D.6.2.a
	C3 2	Holistic	D.7.a, D.7.b, D.7.c
	C4 2	Holistic	D.8.a, D.8.b
L2. Knows how (0.878/0.864–0.874) 1	C1 2	Holistic	D.2.c
	C2 2	Environmental	D.3.1.b, D.3.2.b
		Social	D.4.1.b, D.4.2.b
		Economic	D.5.1.b, D.5.2.b
		Holistic	D.6.1.b, D.6.1.c, D.6.1.d, D6.2.b
	C3 2	Holistic	D.7.d
	C4 2	Holistic	D.8.c
L3. Demonstrates (0.908/0.896–0.908) 1	C1 2	Holistic	D.2.d
	C2 2	Environmental	D.3.1.c, D.3.2.c
		Social	D.4.1.c, D.4.2.c
		Economic	D.5.1.c, D.5.2.c
		Holistic	D.6.1.e, D.6.1.f, D.6.2.c
	C3 2	Holistic	D.7.e, D.7.f
	C4 2	Holistic	D.8.d
L4. Does (0.842/0.805–0.835) 1	C1 2	Holistic	D.9.a
	C2 2	Environmental	D.9.b
		Social	D.9.c
		Economic	D.9.d
		Holistic	D.9.e
	C3 2	Holistic	D.9.f
	C4 2	Holistic	D.9.g
L5. Is (0.656/0.589–0.656) 1	C1 2	Holistic	D.10.a
	C2 2	Environmental	D.10.c
		Social	D.10.d
		Economic	D.10.e
		Holistic	D.10.b
	C3 2	Holistic	D.10.f
	C4 2	Holistic	D.10.g

¹ Results of the analysis of the internal consistency of the scales that constitute the construct and each dimension (Scale Cronbach’s alfa/Scale alfa range if an item is removed, see Section 2.3). ² C1. Competence in the critical contextualization of knowledge through the linking of social, economic, and environmental issues on a local and/or global level; C2. Competence in the sustainable use of resources and in the prevention of negative impacts on natural and social environments; C3. Competence to participate in community processes that promote sustainability; and C4. Competence to apply ethical principles related to sustainability values in personal and professional behavior.

) show results below 0.7 (0.660 and 0.656, respectively). Nonetheless, it is important to indicate that the result of the dimension C1 is directly connected with L5, as the Cronbach’s alfa value of the scale C1 would be 0.708 if the indicator of the level of domain L5 was removed. Nevertheless, due to this reason, and to the fact that these two scales do not have many items (7 each one), their Cronbach’s alfa values were considered acceptable, as proposed by some authors [62,138,139], especially in scales with less than 10 items [140].

4. Discussion

Generalist Architectural Education has been assessed—in Spain and worldwide—to ensure that future architects are capable of practice in line with sustainability. Nevertheless, in relation to the learning experience and outcomes, no comprehensive and systematic tools or approaches exist in AE to jointly or separately assess them from the SD perspective. As a response to this situation and to facilitate the informed adjustment of Spanish AESD by education managers, teachers, policy-makers, and professional associations, the authors have investigated which constructs, dimensions, subdimensions, indicators, items, and variables could be used to assess the level of integration of sustainable development in architectural education and how they could be included in a comprehensive questionnaire addressed to students from the Spanish qualifying Master’s Degree in Architecture. In addition, the production and validation of this prototypical questionnaire were presented in the paper, confirming its validity and preparing the way for its future distribution in different architectural schools.

In particular, the questionnaire will allow for comprehensively and simultaneously measuring the students’ AESD Learning Outcomes, their AESD Learning Experience, and the connection between these two constructs, completing the gap existing in their assessment. Moreover, the questionnaire will allow for studying how the students’ academic, professional, and personal background may influence their answers. The questionnaire has been designed to enable the correlational study between all the considered constructs, dimensions, subdimensions, indicators, and variables. In addition, the questionnaire can be used for longitudinal and/or transversal studies, assessing respectively the evolution in time or evaluating AE at a moment in time, and for studying only one case or multiple cases.

4.1. AESD Learning Outcomes Assessment

The AESD Learning Outcomes have been defined and a scale for surveying the students’ perspective towards their level of competence has been developed. In particular, the sustainability competency map for generalist architecture degrees developed (see Appendix B), based on the results from the EDINSOST project [92], facilitated the definition of a comprehensive collection of sustainability learning outcomes required in AE to allow architects to practice in line with sustainable development. On the one hand, asking the students to assess their learning outcomes, which has been demonstrated as a useful and reliable tool to assess the actual outcomes acquired [77,148], provides a broad perspective of the scope of their education, complementing existing works, which have focused only on competencies related to design or building design. In addition, organizing the outcomes according to the CRUE four “cross-curricular skills for sustainability” permitted the concise address of the cross-cutting competencies for sustainability to complement the existing evaluation of AESD, which has ignored them in all cases, as well as the holistic perspective of SD, and the environmental, social and economic dimensions when further detail was needed, contrary to prior works that only occasionally considered social and/or economic dimensions of sustainability [28,43,53,61]. Moreover, organizing the outcomes according to a taxonomy allowed covering all the levels of domain, which has only been done in one existing work [56,57,58]. In this sense, unlike the EDINSOST project, which considered together Miller’s levels L3 and L4 (“demonstrate” and “do”, respectively) [77], this work has included separately the four levels from Miller’s pyramid (“knows”, “knows how”, “demonstrates” and “does”), as it was found essential to differentiate between what the student believes himself or herself able to do (L3), and what he or she actually does or has done (L4). Additionally, based on the work from Delors et al. [74] that defined The Four Pillars of Learning (“learning to know”, “learning to live together”, “learning to do”, and “learning to be”), the AKASA model used before in the AESD assessment [56,57,58], and Azjen’s Theory of Planned Behavior [144,145], who stated a clear difference between behaviors and attitudes, it was found was essential to add a fifth level (L5. Is) for defining the students’ attitudes, a key domain for enabling ESD [75,143]. Despite the significance of the “AESD Learning Outcomes” construct, it is important to state that separating Miller’s levels L3 and L4, and adding a fifth one regarding the students’ attitudes (L5), made the questionnaire longer than the reference ones. Nonetheless, its length did not seem to significatively affect the quality of the responses to the test of the final version (see Section 2.3), as the low and homogeneous proportion of “Don’t know” answers has shown (see Section 3.8), may be due to the variation in the type of questions and answers in levels L4 and L5, which entail a change in routine that grabs the participants’ attention at the end of this part of the questionnaire (D).

4.2. AESD Learning Experience Assessment

The AESD Learning Experience has been conceptualized and operationalized, and a scale for surveying the students’ perspective towards it has been designed based on the existing theoretical background and two reference tools [85,86,87,88,89,90,91]. Therefore, regarding formal education, it was possible to go beyond the assessment of the contents and their organization from existing works to cover also the methods and assessment, which only a few works have done before in AE [28,32,33,34,40,43,44,45,46,47,49,52,55], as well as the cross-cutting competencies for sustainability and the different levels of domain that had been ignored in all cases, and a comprehensive perspective of SD, covered before in only a few cases [27,28,31,43,49,53,55]. Furthermore, apart from formal learning, this new tool permits addressing non-formal and informal learning, which are fundamental components of ESD [6,65,66,67,68,69,70,71] yet overlooked by all prior works except one [55], which covered non-formal learning. Although the questionnaires used as a reference were addressed to teachers [85,86,87,88,89,90,91], here students were found as relevant and reliable informers, as they had been in prior evaluations of AESD [28,43,49,53,54], allowing us to assess all at once the learning outcomes and experience. Moreover, to let the students understand the education-related concepts that may be unclear for them, it was necessary to divide and detail more general items. Likewise, the original dimensions and items were completed with perspectives that were found essential in the complementary literature [6,24,32,64,65,67,68,69,70,71,75,76,109,110,111,112,113,114,115], especially concerning non-formal and informal learning supported by the centers, which had been ignored in the questionnaires used as a reference [85,86,87,88,89,90,91]. In this sense, it is important to highlight that this construct received the highest level of “Don’t know” responses in the final version test (see Section 3.8), although some of them may have been caused by the students’ lack of awareness towards the reality being asked. This fact, together with the aim of addressing all components of formal learning, as well as non-formal and informal aspects, increased the number of items in relation to other questionnaires. Nevertheless, depending on the sample to which the questionnaire will be addressed, the conditions of distribution and/or the level of detail aimed, the possibility of simplifying the structure behind the “AESD Learning Experience” construct could be considered, by grouping some indicators and keeping the dimensions and subdimensions as they already are.

4.3. “AESD Learning Experience” Contribution to “AESD Learning Outcomes” Acquisition Assessment

Finally, the “AESD Learning Experience contribution to AESD Learning Outcomes acquisition” has been conceptualized and operationalized, and a scale for surveying the students’ perspective towards it has been designed. Although this relationship is going to be studied through the correlation of the answers related to the prior constructs, and that no existing work assessed it through specific questions, it was considered important to explicitly assess the students’ point of view about it to contrast the correlational approach. In particular, this construct includes a comprehensive perspective towards the acquisition of all learning outcomes. Nonetheless, due to this overall view, and according to the feedback from the focus group with participants in the pilot test of the second draft, it was found difficult for students to assess all at once the 5 levels of domain (L1–L5), as well as to differentiate between L1, L2, and L3. Therefore, L1, L2, and L3 were assessed together and L4 and L5 separately, structuring the construct in 3 indicators and items.

4.4. Students’ Academic, Professional, and Personal Background Survey

Apart from the prior constructs, different properties regarding the students’ backgrounds were included. These properties were selected to control any significant influence that they may have on the students’ perspective towards the questionnaire constructs (e.g., if they studied the Bachelor’s and the Master’s degree in the same school). Moreover, some of them were found necessary for sampling or for assessing the sample representativeness. Although this high number of properties made the questionnaire larger than usual, it provides a rich and varied perspective about the students’ background.

4.5. Questionnaire Quality

Apart from the revision by experts of the first draft of the questionnaire and the pilot test of the second one (see Section 2.2.2 and Section 2.2.4, respectively), the analysis of the answers to the final version from 261 students from the qualifying Master’s Degree in Architecture from 19 Spanish schools has demonstrated the consistency and reliability of the results that can be obtained by applying the questionnaire it (see Section 3.8).

4.6. Questionnaire Transferability

The questionnaire presented in this work has been designed and validated for evaluating the Spanish qualifying academic program conformed by the Bachelor’s Degree in Architecture and the Master’s Degree in Architecture from different schools, by developing and testing it with different groups of experts and students. Nonetheless, it was designed so that it could be used with students from any Bachelor’s and/or Master’s degree in Architecture aimed to prepare them for professional practice as generalists. Thus, it could be addressed to students from degrees under the umbrella of the United Nations Educational, Scientific and Cultural Organization and the International Union of Architects (UNESCO-UIA) validation system, the European Union (EU) Directive for Architects, the Royal Institute of British Architects (RIBA) Parts 1 and 2 validation, the National Architectural Accrediting Board (NAAB) accreditation—from the United States—, or their equivalents, with a little adjustment of degree-specific questions. Furthermore, its structure (constructs, dimensions, and subdimensions) could be directly applied, and its contents (items, variables, and values) easily adjusted, to assess HE in other disciplines or from the perspective of other agents (education managers, teachers, etc.), in the same way that the tools used as a reference for designing this questionnaire have been adapted in this work to AE and/or the students’ viewpoint [13,77,82,85,86,87,88,89,90,91,92,93,94,95].

4.7. Questionnaire Length

As it has been stated in prior paragraphs, it is important to emphasize that the questionnaire is larger than usual. In particular, despite the low rates of “Don’t know” responses obtained throughout the whole questionnaire and at the end of each section in the test of the final version, in our experience, it should be kept in mind that 20 students (out of 281) stopped doing the questionnaire between sections D and E when doing it at home on their own (see Section 3.8.3). Therefore, it is recommended to complete the questionnaire during school time. Alternatively, it can be completed in two stages (stopping between parts D and E) or, if it is aimed only to assess the learning outcomes or the learning experience, it can be used in parts to considering only the students’ outcomes (parts A, B, C, D, F, and G) or learning experience (parts A, B, C, E, F, and G).

4.8. Future Lines of Research

Below are details of future lines of research that arose from this work:

• Use of the questionnaire in different schools of architecture from Spain;
• Triangulation of results with other assessments of AESD (regarding curricula or concerning complementary approaches or perspectives, such as documentary analysis, interviews or seminars, or the education managers’, the pedagogical experts’, the teachers’ or the point of view of education managers, pedagogical experts, teachers or practitioners);
• Adaptation of the questionnaire to other generalist architecture degrees;
• Adaptation of the questionnaire to other HE degrees unrelated to generalist architecture.

5. Conclusions

Generalist Architectural Education has been assessed from the sustainability point of view—worldwide and in Spain—to ensure that future architects are capable of practicing in line with SD, regarding the curriculum of degrees, the learning experience provided by schools, and the learning outcomes acquired by students. Nevertheless, the existing evaluation shows a big gap between the factors that have been assessed and those proposed by the literature on ESD and AESD for a thorough understanding. Concerning the learning experience and outcomes, despite the preference for assessing them through questionnaires for students and/or teachers or education managers, no comprehensive and systematic tools or approaches exist in AE to jointly or separately do so. On the contrary to AE, other disciplines have evaluated in Spain and worldwide the students’ learning experience and outcomes separately with comprehensive tools that provide valuable inputs for evaluating simultaneously the AESD learning experience and outcomes through the students’ perspective, with relevant works regarding the assessment of the learning outcomes acquired by students, such as the questionnaire from the EDINSOST Project, and the evaluation of the learning experience provided by schools, such as the questionnaires Autodiagnóstico del Profesorado en Sostenibilización Curricular APROSOS and Sostenibilidad y Práctica Docente. As a response to this situation and in line with the aforementioned works, a close-ended questionnaire for students from the Spanish qualifying Master’s Degree in Architecture has been designed and validated to facilitate the adjustment of AE in Spain towards the introduction or enhancement of SD by education managers, teachers, policymakers, and professional associations. This questionnaire allows for comprehensively and simultaneously evaluating the students’ perspective of their learning outcomes related to AESD; their learning experience regarding formal, non-formal, and informal AESD; and the connection between; as no other tool has done before in the field of AE or other HE. This information, together with curricula assessment, the perspective of other AE agents, additional approaches, and further research directions, are considered essential to inform the future adjustment of Spanish AE towards AESD. Thus, systematic, homogeneous, and comparable longitudinal and cross-sectional assessments of single or multiple generalist architecture degrees from Spain can and should be performed. In particular, the validation of the questionnaire and the assessment of the quality of the answers from 261 students from different schools have confirmed the consistency and reliability of the results that can be obtained. Furthermore, this tool could be adapted to evaluate similar degrees around the globe and its structure and contents directly applied or easily adjusted to assess HE in other disciplines or from the perspective of other agents. Therefore, since only taking informed actions on AESD future architects will be able to contribute to sustainable development at their practice, this work aims to constitute the first step towards that objective in Spanish AE.