Validation and Standardization of a Questionnaire for the Self-Assessment of Service-Learning Experiences in Higher Education (QaSLu-27)

Abstract

There has been a proliferation of service-learning practices in many disciplines of higher education. Although there are many instruments to assess various aspects related to service-learning, only a few are valid and reliable. This research tries to fill that gap by providing an optimal instrument. To this end, 118 educators from 43 higher education institutions participated in the validation process. Sampling was incidental and based on opportunity. Participants were asked to fill in the Questionnaire for the Self-assessment of University Service-learning Experiences—45 (QaSLu-45), which consists of 45 items with a 5-point Likert scale and was validated qualitatively through the Delphi method in 2020. To validate the questionnaire psychometrically, four steps were followed. First, an exploratory factor analysis of the main components was performed. Next, robust unweighted least squares factor exploratory analysis was carried out. Then, the goodness-of-fit and reliability of the final questionnaire were calculated. Finally, the scales for the new reduced QaSLu-27 were established. These results provide a valid, robust, reliable instrument, the QaSLu-27, which not only facilitates the evaluation of service-learning experiences for improving teaching and learning processes but can also be useful for their design.

1. Introduction

Service-learning is an experiential educational method in which students participate in activities that address human, social, and environmental needs from the perspective of social justice. It integrates service to the community with the academic curriculum. This enriches learning in an academic program, fosters civic responsibility, and strengthens communities through action and critical reflection [1].

It is an educational process that engages students by improving their understanding of academic concepts through practical application, thereby improving the academic commitment of university students [2]. It is also an instrument of social transformation that promotes sustainable development and social justice [3]. It is based on practical experience, and it guides students to link academic knowledge with community needs, with the aim of fostering civic engagement through reflective social action [4,5,6]. Service-learning goes beyond the boundaries of the classroom, bringing students into the community, with reciprocal benefits for both the students and the community [4,7,8].

Although it is not new, service-learning has, in recent years, spread rapidly throughout the world of higher education [9,10]. As a result, legislation that regulates higher education frequently recognizes the value of universities in the construction of global citizenship. For example, in the European framework, the European Commission’s Renewed Agenda for Higher Education [11] encourages universities to develop their civic function and emphasizes the importance of working with the community.

In this framework, the need for validated instruments that evaluate the quality of university educational proposals based on service-learning [12] becomes evident. Some educational experiences are called service-learning, even though they do not meet the standards of this methodology; in some cases, what is called “service-learning” is more about volunteering activities or field practices [13]. For this reason, instruments are needed that respect the pedagogical bases of service-learning in the design, preparation, implementation, and evaluation of university programs [14]. Currently, most of this information is collected through students [15]; it would be interesting to gain the perspective of educators.

The questionnaire provided is based on a broader questionnaire that underwent robust unweighted least squares (RULS) exploratory factor analysis. This analytical approach is relatively new, with only one comparable study identified [2]. Notably, the QaSLu-27 questionnaire is shorter than its predecessor, yet it maintains its reliability. Hence, the reduction in size of the measurement tool does not compromise its quality. Furthermore, it introduces scales, enabling educators to evaluate their service-learning experiences based on their own traits and on those of the service-learning context, a feature previously unavailable.

In this article, the process for presenting the validation and standardization of a questionnaire for the self-assessment of service-learning experiences in higher education (QaSLu-27) is outlined. The section “Evaluation of Service-learning in Higher Education” reviews both non-validated and validated instruments developed to ensure the quality of this experiential educational method. Subsequently, it clarifies the objectives of this study. Following this, the Section 4 provides comprehensive details on participant selection, data collection instruments, procedures, and ethical considerations. Subsequent sections present the results obtained from various analyses, followed by discussions juxtaposing these findings with other studies to draw meaningful conclusions.

2. Evaluation of Service-Learning in Higher Education

2.1. Non-Validated Instruments

The scientific community has various instruments available to assess the quality of service-learning experiences in higher education. While these instruments are interesting, their validity and reliability have not been studied in depth.

One of the most used instruments is the rubric. For example, it is worth noting the self-assessment rubric for the implementation of service-learning in higher education by Furco [16], structured in five dimensions: (1) university mission and philosophy; (2) academic support for involvement in civic engagement; (3) student support for and involvement in civic engagement; (4) community participation and partnerships; (5) institutional support for service-learning and national debate, policy, and support.

If the objective is to evaluate the quality of university pedagogical proposals based on service-learning, there are several options. On the one hand, Martín et al. [14] created a rubric to evaluate and improve the formative quality of service-learning experiences in higher education based on the fact that the optimal design of pedagogical practices has a direct effect on students’ learning.

On the other hand, Lorenzo and Belando-Montoro [17] delved into the criteria and quality indicators to evaluate service-learning experiences in higher education and published a four-scale questionnaire featuring (1) civic development, (2) participation and commitment in the community, (3) vocational and professional development, and (4) personal development. Even today, to respond to the situation caused by the COVID-19 virus, Shek et al. [18] present us with a proposal for evaluating the quality of digital service-learning experiences, using multiple evaluation strategies to assess the effectiveness of these experiences.

Finally, other authors focused on the development or impact that service-learning has on the students who perform the service. Lu and Lambright [19] proposed a scale to find the opinion of higher education students about which factors influence their service-learning outcomes.

2.2. Validated Instruments

Although different instruments to assess various aspects of service-learning are frequently used, not many are valid and reliable.

Among these instruments, it is worth mentioning those that evaluate the quality of service-learning experiences designed and implemented in higher education. For example, the questionnaire developed by Escofet-Roig et al. [20], consisting of 16 items, assesses service-learning experiences from the perspective of university students, encompassing three dimensions: participation, service, and competencies. A fourth dimension is added transversally, which refers to general satisfaction with participation in the experience. All 16 items comply with the requirement of having the shortest possible form, and they follow the “funnel” procedure, moving from the most general items to the most specific ones.

Content validity was conducted by a group of experts in the area (validation through expert judgment). Eight educators from different disciplines who worked in service-learning participated. With the changes suggested by the judges, a second version of the instrument was prepared.

The second version underwent an empirical validation process to test reliability. A pilot test was carried out, and the questionnaire was applied to a total of 116 university students who participated in service-learning experiences (84.5% female and 15.5% male). For the reliability of the instrument, analysis of the internal consistency of the items was followed through the calculation of Cronbach’s alpha coefficients. The index obtained in all the items indicated a high correlation; therefore, the reliability was acceptable. The global Cronbach’s alpha was around 0.90. The contribution of each item to its respective scale (corrected homogeneity index, which provides discrimination capacity) was, in all cases, a positive value.

Another validated instrument is the scale to measure the impact of service-learning experiences on the development of the professional competencies of student teachers, designed and validated by Rodríguez-Izquierdo [21]. This scale is structured into five factors: (1) ethical commitment, (2) cooperation with other professionals, (3) design and development of experiences, (4) readiness for diversity, and (5) readiness for professional development. In total, 366 Social Studies students participated. For the selection of the sample, a simple, stratified, multistage random sampling was carried out.

To strengthen the validity of the content, it was submitted for evaluation to nine expert judges. With the suggestions received, the items were reduced and/or adjusted according to the validity criteria, location, intelligibility, and univocity. As a result, an instrument with two sections was configured, the first with demographic information and the second with a Likert-type scale of five points and 30 items, in which students assessed to what extent service-learning favors the development of professional competencies.

To verify the validity of the construct, an exploratory factorial analysis was carried out following the principal component extraction method (Varimax rotation). In addition, different confirmatory factorial analyses (CFAs) were used under the maximum likelihood criterion for parameter estimation.

The scale is a valid and reliable instrument to assess professional competencies. The Cronbach’s alpha obtained amounts to 0.87 for the entire instrument, as well as in each of the dimensions, which ranged between 0.84 and 0.91. The fit of the CFA was highly satisfactory (χ² = 881.22, p = 0.00; GFI = 0.93; CFI = 0.98; SRMR = 0.067; RMSEA = 0.064). Among the limitations of Cronbach’s alpha, the author points out that it was carried out with a sample of students from a single educational program, and it features a quasi-experimental design that does not allow the establishment of causal relationships.

A year later, the same author, Rodríguez-Izquierdo [2], validated another instrument named the Utrecht Work Engagement Scale for Students (UWES-S-9). It is composed of nine items grouped into three dimensions: (a) vigor: level of energy, persistence, and effort in performing academic tasks; (b) dedication: high level of involvement in studies and with their career; and (c) absorption: high level of concentration and immersion in what they do when they study. All these items are scored on a 7-point Likert scale from 0 = not at all/never to 6 = every day/always.

The sample consists of 342 students, of which 183 have a bachelor’s degree in Social Studies, and 153 have a double degree in Social Studies and Social Work from Pablo de Olavide University (UPO) in Seville, Spain. A random, stratified, multistage sampling was carried out, with strata established according to the students’ most relevant characteristics: gender, year, age, and the route of entry to university.

The internal structure of the evaluation instrument was corroborated by dividing the sample into two randomly drawn subsamples. To determine the number of factors, an exploratory factor analysis was performed on the first half of the sample (n1 = 178). The results indicated that the instrument responded to a one-dimensional structure (ECV values were between 0.70 and 0.85; MIREAL values were less than 30). A CFA was performed on the second half of the sample (n2 = 164). The results were highly satisfactory (S-B2 values were greater than 0.01; NNFI and CFI values were equal to or greater than 0.95; SRMR and RMSEA values were less than 0.08).

For reliability, composite reliability (CR), and maximum reliability (MR), Cronbach’s alpha and McDonald’s omega were calculated, and the instrument obtained satisfactory results. Finally, discriminant validity was calculated with adequate outcomes.

There is also an instrument designed and validated by León-Carrascosa et al. [9] to evaluate the educational value of service-learning in higher education. This instrument has three basic dimensions: training (as an objective), learning (as a means), and service (as a commitment to the community).

The study was carried out with a sample of 180 students from 9 Spanish universities from the regions of Catalonia, Galicia, Valencia, and Madrid. Incidental non-probabilistic sampling was carried out.

The instrument is made up of 35 items, and its objective is to evaluate the service-learning methodology of its protagonists. Students respond on a Likert-type scale of 1 to 5.

The content validity of the instrument was carried out by expert judges in educational research and experts in service-learning in higher education. The internal consistency study of the instrument is excellent, as the value of Cronbach’s alpha is 0.95. Reliability was studied after the CFA, obtaining an excellent final value and satisfactory values by dimensions (α Global = 0.95, α Formative Dimension = 0.88, α Learning Dimension = 0.90, α Service Dimension = 0.91).

Another noteworthy contribution is the scale designed and validated by Santos-Pastor et al. [22] to measure the perception of higher education students about the impact that participation in service-learning experiences has had on their learning and on their personal and social development. This instrument has 41 items and one open question, and it allows for objectifying and validating the concrete effects that this pedagogical approach has in its formative, professional, personal, and community dimensions. Its validation included 200 students from five Spanish universities who had participated in different service-learning experiences in Physical Activity with disadvantaged groups. A convenience sampling that adjusts to the parameters established by Nunnally was used. The content validation was carried out through the critical judgment of six experts in university service-learning, both from the pedagogical and Physical Activity fields.

A confirmatory factor analysis was performed to assess the structure of theoretical dimensions and the internal consistency of the global scale, and each of its dimensions was verified. The results confirm the adequacy of the chosen structural model (RMSEA = 0.08) and the reliability of the scale (α = 0.95) and of each of its dimensions (α between 0.68 and 0.86). Cronbach’s alpha was used to analyze the internal consistency of the scale, which was also calculated for each of its dimensions. The value of Cronbach’s alpha for the global E-ASAF scale was α = 0.95.

The main limitation of this study was the selection of the participants, who came from a convenience sample of students who participated in service-learning experiences related to physical education, as there are few of these in the university context.

The psychometric validation of the VAL-U instrument developed by Ruiz-Ordoñez et al. [23] was also evaluated. This instrument emerges as a response to the need to develop instruments for measuring the impact that service-learning methodology has on the values and civic attitudes of higher education students. It consists of 20 items that are answered through a Likert scale of 5 points. The authors analyzed the internal consistency and factorial structure of the instrument with data from 162 university students. The scale confirmed acceptable psychometric properties. Cronbach’s alpha was 0.67, and 3 factors emerged from the factor analysis.

In addition, it is worth considering the contribution of Gul et al. [15], who developed and validated an instrument to measure service-learning management. The sample for validation was made up of 315 educators. Items were selected based on the deductive method using the existing theory. Exploratory factor analysis (EFA) was used to explore the factor structure of the scale. A four-factor structure with 21 items was identified. The reliability of this scale is very high since a Cronbach’s alpha of 0.96 was obtained.

Finally, a study that attempts to respond to the scarcity of tools for guiding their development and for assessing the quality of university service-learning experiences has recently been published [24]. The authors developed criteria using the Delphi method. It was built as a matrix with 9 dimensions and 43 indicators. The dimensions are organized into two main blocks: phases (diagnosis, design, intervention, evaluation, and reflection) and agents involved (target group or social challenge, students, educators, and organizations). As this instrument has not yet been used, its reliability and psychometric validity have not been tested.

3. Objective of the Study

While there some are instruments to evaluate service-learning experiences in higher education, they are rare. Moreover, few have undergone qualitative validation before psychometric validation. In addition to quantitative validations, most of the analyses explore the factorial structure of the designed instruments without optimizing their final structure.

Analysis of previous literature reflects the need to develop new instruments to evaluate service-learning experiences in higher education, optimizing their design qualitatively and quantitatively. This study aims to validate and establish the scales of a Questionnaire for the Self-assessment of University Service-learning Experiences (QaSLu). QaSLu is an abbreviation of “Questionnaire for Self-Assessment of Service-Learning Experiences in University”. The authors use “university” as a synonym for “higher education”. The title of the questionnaire in Spanish is “Cuestionario para la Autoevaluación de Experiencias de Aprendizaje-Servicio Universitario”.

4. Materials and Methods

4.1. Selection of Participants

Sampling was incidental and based on opportunity [25]. To configure the sample, all the proceedings of conferences organized by the Spanish Network of University Service-learning that were published up to 2018 were reviewed. An email was sent to all the authors, inviting them to participate in the study. The email asked the authors to forward it to contacts who might be interested in participating. Therefore, incidental sampling was complemented by snowball sampling [26].

In total, 118 educators from 43 higher education institutions participated, of which 67.80% were women and 32.20% were men. The mean age was 46.16 years (SD = 9.60), with the minimum age being 24 and the maximum being 65. Figure 1 reflects the distribution of this sample, taking into account gender and age in a population pyramid.

Regarding the academic level of the participants, a large majority (76.3%) had a doctoral degree, while the rest had at least a bachelor’s degree. Regarding their experience in using service-learning, the average was 6.69 years (DT = 5.39), with a half-year at the lowest and 34 years at the highest. In Figure 2, this distribution can also be observed according to gender.

The institutions with which the universities collaborated are the following: educational centers (including other universities), social centers, centers for older people, health centers, NGOs, foundations, and government agencies such as town halls, councils, and so forth. Partnerships for developing service-learning were established with two categories of institution: formal education centers, which account for 50.85% of the sample, and centers not linked to formal education, which account for the other 49.15%.

Concerning the modality of the service-learning experiences evaluated, 64.41% were face-to-face, while 35.59% included some type of virtual activity.

4.2. Instrument for Collecting Information

Participants were asked to fill in the QaSLu-45 (45-item version of the Questionnaire for Self-Assessment of Service-Learning Experiences in University). QaSLu-45 (available for download as Supplementary Material A for non-commercial use) was validated through the Delphi method [12]. Its objective is to assess the quality of the service-learning experiences in higher education. The questionnaire consists of 45 items with a 5-point Likert scale. It is structured as follows: (1) purpose of the questionnaire, written informed consent, and instructions; (2) institution or organization, age, gender, level of education, and years of professional experience in service-learning of the respondent (this information does not make it possible to identify the participants); (3) data on the service-learning experience; (4) 45 items for self-assessment of service-learning experiences (5) appreciation for the participation.

4.3. Procedure

The study consisted of four phases. The objective of the first phase, which began in September 2019, was to identify and configure the sample. The second phase was intended to make the call to participate in the study by asking the selected persons to complete QaSLu-45. Due to the global health emergency caused by the SARS-CoV-2 virus, this phase was extended until January 2021. The third phase was undertaken throughout 2021; a descriptive study of the responses obtained was carried out to validate the questionnaire. The first step for validation was to perform an exploratory factor analysis of the main components through the statistical software IBM SPSS (version 25.0). To check if this analysis was optimal, following the recommendations of López-Aguado and Gutiérrez-Provecho [27], the Bartlett Sphericity Test and the Kaiser–Meyer–Olkin Sampling Adequacy Test were performed [28]. The second step consisted of carrying out a RULS factor exploratory analysis [29] through the FACTOR software (version 12), designed by Ferrando and Lorenzo-Seva [30], and for which they determined the criteria to be followed. In this analysis, the principle of parsimony was followed, allowing the configuration of an optimized and robust design that provides a unique, valid, reliable score. In addition, in the third step, the reliability of the final questionnaire was calculated through the Model Fit Measures of CFA and Cronbach’s alpha with the statistical software IBM SPSS (version 25.0). In the fourth and final step, the scales for the new and reduced QaSLu-27 (27-item version of Questionnaire for Self-Assessment of Service-Learning Experiences in University) were established based on the gender and age of the teaching staff, the gender and experience in service-learning by the teaching staff, the gender and type of institution in which the service-learning is developed, and the gender and nature (virtual or face-to-face) of the service-learning practice. These scales were calculated taking into account the percentile distribution of each item according to the participants of the study.

QaSLu-27 is available for download as Supplementary Material B for non-commercial use.

4.4. Ethical Considerations

As explained in the description of the instrument, the personal information collected through the questionnaire does not allow identification of the participants, so the questionnaire guarantees the anonymity of the participants, according to the Spanish Organic Law 3/2018, of 5 December 2018, on Data Protection and Guarantee of Digital Rights.

5. Results

This section consists of two subsections. The first presents the analyses carried out to ensure the validity, robustness, and reliability of the QaSLu-45. The second presents the scales established for the use of the QaSLu-45.

5.1. Validity of the QASLU-45 (Available for Download as Supplementary Material A for Non-Commercial Use)

The quantitative validation is presented in

Table 1. Univariate descriptions of QaSLu-45 items.

	Media	C.I. (95%)	Variance	Asym.	Kurtosis		Media	C.I. (95%)		Variance	Asym.	Kurtosis
1	2.941	(2.66 3.22)	1.429	−0.786	−0.554	24	3.059	(2.86 3.25)		0.683	−0.657	−0.039
2	3.322	(3.07 3.58)	1.167	−1.443	0.847	25	3.695	(3.51 3.88)		0.602	−3.032	9.345
3	3.364	(3.16 3.57)	0.740	−1.506	2.475	26	3.525	(3.35 3.70)		0.521	−1.455	1.468
4	2.958	(2.73 3.18)	0.888	−0.712	−0.051	27	3.424	(3.24 3.61)		0.617	−1.332	1.754
5	3.432	(3.23 3.63)	0.703	−1.654	2.623	28	0.907	(0.67 1.14)		1.000	0.907	−0.055
6	0.263	(0.09 0.43)	0.516	3.436	12.642	29	3.364	(3.18 3.55)		0.588	−1.300	2.237
7	3.280	(3.09 3.47)	0.659	−1.227	1.796	30	2.814	(2.54 3.08)		1.304	−0.764	−0.154
8	2.915	(2.64 3.20)	1.400	−0.764	−0.431	31	3.297	(3.09 3.50)		0.768	−1.152	0.882
9	2.907	(2.66 3.16)	1.101	−0.877	0.269	32	2.619	(2.35 2.89)		1.287	−0.613	−0.304
10	3.449	(3.27 3.63)	0.586	−1.655	3.379	33	2.941	(2.69 3.19)		1.124	−0.613	−0.550
11	3.390	(3.19 3.59)	0.712	−1.525	2.219	34	2.966	(2.72 3.22)		1.117	−0.800	−0.070
12	3.534	(3.35 3.72)	0.588	−1.935	4.142	35	1.373	(1.10 1.64)		1.302	0.438	−0.770
13	3.280	(3.07 3.49)	0.812	−1.566	2.828	36	2.958	(2.70 3.21)		1.159	−0.819	−0.130
14	3.551	(3.38 3.72)	0.519	−1.827	3.314	37	3.305	(3.11 3.50)		0.704	−1.062	0.386
15	3.364	(3.16 3.57)	0.740	−1.264	1.129	38	2.686	(2.41 2.96)		1.368	−0.617	−0.406
16	2.229	(1.99 2.47)	1.041	−0.328	−0.196	39	3.110	(2.87 3.35)		1.030	−0.910	0.054
17	3.432	(3.25 3.61)	0.567	−1.147	0.530	40	3.186	(2.96 3.41)		0.914	−1.321	1.642
18	2.831	(2.57 3.09)	1.226	−0.642	−0.371	41	3.381	(3.22 3.54)		0.473	−0.983	0.926
19	3.475	(3.26 3.69)	0.809	−2.074	4.346	42	2.636	(2.42 2.85)		0.808	−0.413	0.377
20	3.229	(3.00 3.46)	0.922	−1.169	0.611	43	2.602	(2.40 2.80)		0.731	0.049	−0.289
21	3.466	(3.27 3.66)	0.673	−1.982	4.605	44	2.636	(2.38 2.89)		1.164	−0.335	−0.691
22	2.992	(2.74 3.24)	1.093	−0.925	0.381	45	2.881	(2.63 3.13)		1.105	−0.688	−0.193
23	2.398	(2.05 2.75)	2.189	−0.362	−1.318

C.I. (95%): 95% confidence interval/Asym.: asymmetry.

, based on the variance distribution of each item.

For this validation, first of all, an exploratory factor analysis of the principal components was carried out. The purpose of this was to analyze the matrix structure of the items so that the main components, which explain a large part of their variance, could be found.

The significance of Bartlett’s Sphericity Test (χ² = 2689.72; p = 0.00) and that of the Kaiser–Meyer–Olkin Sampling Adequacy Test (KMO = 0.793) confirm that this factorial analysis is optimal due to the good design of the items and the sampling for the validation of QaSLu-45. Regarding the reliability of QaSLu-45, these 45 items jointly showed excellent reliability measured through Cronbach’s alpha (α = 0.90) and an excellent robust goodness-of-fit measured through the model fit measures of CFA since the minimum discrepancy of CFA/degrees of freedom is between 1 and 3 (CMIN/DF = 1.053), the Comparative Fit Index is >0.95 (CFI = 0.973), following Kelley’s proposed criterion [31] the weighted root mean square residual is <1.0 (WRMR = 0.0978), and the root mean square error of approximation is <0.06 (RMSEA = 0.042) [32].

The results of the principal component factor analysis show 13 components that exceed an eigenvalue of 1.0, explaining 69.95% of the total variance (see

Table 2. Variance of the 45 items explained through the components with an eigenvalue greater than 1.

Component	Initial Eigenvalues			Extraction Sums of Squared Loadings
	Total	Variance %	Cumulative %	Total	Variance %	Cumulative %
1	11.817	26.261	26.261	11.817	26.261	26.261
2	2.456	5.459	31.719	2.456	5.459	31.719
3	2.373	5.273	36.992	2.373	5.273	36.992
4	2.081	4.624	41.616	2.081	4.624	41.616
5	1.914	4.254	45.870	1.914	4.254	45.870
6	1.643	3.651	49.521	1.643	3.651	49.521
7	1.552	3.449	52.970	1.552	3.449	52.970
8	1.458	3.240	56.211	1.458	3.240	56.211
9	1.435	3.189	59.400	1.435	3.189	59.400
10	1.319	2.931	62.331	1.319	2.931	62.331
11	1.258	2.795	65.125	1.258	2.795	65.125
12	1.112	2.470	67.596	1.112	2.470	67.596
13	1.061	2.358	69.953	1.061	2.358	69.953

). However, the sedimentation graph generated (see Figure 3) allows verification of how the relevance of the first component is so disproportionate that the relevance of the rest of the components is inestimable; therefore, it is advisable to reduce the QaSLu-45 spectrum to a single component and to obtain a unique final score with the designed QaSLu-45.

Given these results of the initial factorial exploration, we proceeded to perform a RULS factor exploratory analysis, following the recommendations of Lorenzo-Seva and Ferrando [29], that optimally adjusted the configuration of QaSLu-45 for a single principal component, thereby reducing the excess items according to the principle of parsimony. To do this, as a preliminary step, the score matrix of each item for the main component was analyzed (see

Table 3. Matrix of scores of each item for the main component.

Item	Score	Item	Score	Item	Score
1	0.361	16	−0.110	31	0.464
2	0.501	17	0.553	32	0.576
3	0.583	18	0.270	33	0.555
4	0.448	19	0.496	34	0.590
5	0.680	20	0.403	35	−0.322
6	−0.437	21	0.640	36	0.582
7	0.397	22	0.569	37	0.649
8	0.264	23	0.350	38	0.457
9	0.431	24	0.521	39	0.687
10	0.490	25	0.446	40	0.692
11	0.469	26	0.552	41	0.461
12	0.580	27	0.674	42	0.383
13	0.525	28	−0.590	43	0.521
14	0.549	29	0.530	44	0.526
15	0.570	30	0.490	45	0.493

C.I. (95%): 95% confidence interval/Asym.: asymmetry.

), and the items that had an inverse link with the main component were discarded.

From this first analysis, it was decided to discard items 6, 16, 28, and 35 from the final configuration since they showed an inverse link with the main component.

Using the new optimized version of 41 items, a full factorial analysis was carried out. This included an analysis of the robustness of the principal components for a configuration of a single principal component. For this, following the recommendations of Lorenzo-Seva and Ferrando [33], the Measure of Sampling Adequacy (MSA) was applied, discarding the values of MSA below 0.49. This suggests that, as the item does not measure the same domain as the remaining items in the pool, it should be removed.

This new analysis suggests discarding 14 items to adapt the design to a single principal component: 1, 4, 7, 8, 9, 11, 18, 20, 23, 25, 31, 38, 41, and 42, leaving a final configuration of 27 items, optimized for a single principal component and with a high associated psychometric robustness. This version was called QaSLu-27 (available for download as Supplementary Material B for non-commercial use).

Finally, to determine the improvement in validity, reliability, and robustness of the final QaSLu-27 configuration, a new RULS factor exploratory analysis of the main components was carried out for a final configuration of a single component. With this version of 27 items, there is no need to discard any new item, even increasing the minimum values to 0.50. On the other hand, a considerable improvement was observed, both in the significance of Bartlett’s Sphericity Test (χ² = 1447.76; p = 0.000) and in the Kaiser–Meyer–Olkin Sampling Adequacy Test (KMO = 0.863), which confirm that the robustness and validity of this final configuration is considerably higher than the 45-item design. Regarding the reliability of the QaSLu-27, despite having reduced the number of items from 45 to 27, the reliability measured through Cronbach’s alpha improved (α = 0.92), and the robust goodness-of-fit statistics measured through CFA are slightly better, as can be seen in

Table 4. QaSLu Model Fit Measures.

Measure	QaSLu-45			QaSLu-27
	Estimate	Threshold [31,32]	Interpretation	Estimate	Threshold [31,32]	Interpretation
CMIN	994.830	—	—	332.746	—	--
DF	945.000	—	—	324.000	—	--
CMIN/DF	1.053	Between 1 and 3	Excellent	1027	Between 1 and 3	Excellent
CFI	0.973	>0.95	Excellent	0.979	>0.95	Excellent
WRMR	0.0978	<1.0	Excellent	0.0985	<1.0	Excellent
RMSEA	0.042	<0.06	Excellent	0.046	<0.06	Excellent
Cronbach’s α	0.90	>0.90	Excellent	0.92	>0.90	Excellent

.

It should also be noted that, in this new configuration of 27 items, only 7 components exceed an eigenvalue of 1.0. This explains 65.59% of the total variance (see

Table 5. Variance of the 27 items explained through the components with an eigenvalue greater than 1.

Component	Initial Eigenvalues			Extraction Sums of Squared Loadings
	Total	Variance %	% Accumulated	Total	Variance %	% Accumulated
1	9.072	33.599	33.599	9.072	33.599	33.599
2	1.862	6.897	40.496	1.862	6.897	40.496
3	1.628	6.029	46.525	1.628	6.029	46.525
4	1.568	5.808	52.333	1.568	5.808	52.333
5	1.261	4.671	57.004	1.261	4.671	57.004
6	1.229	4.551	61.555	1.229	4.551	61.555
7	1.090	4.037	65.592	1.090	4.037	65.592

). In the sediment graph generated (see Figure 4), a considerable improvement can be observed in the relevance of the first component, which has gone from explaining 26.26% of the variance to explaining 33.60%. Consequently, it can be clearly stated that this process of item reduction due to factor analysis derived a concise, valid, robust, and reliable QaSLu-27.

5.2. Standardization of the QASLU-27 Scale

Below are the scales established for the QaSLu-27 based on the gender and age of the teaching staff, the gender and experience in service-learning of the teaching staff, the gender and the type of institution in which service-learning is developed, and the gender and nature (virtual or face-to-face) of the service-learning practice (see

Table 6. Gender and age-based scale.

Level	Percentile	Female			Male
		<40 Years	40–49 Years	>50 Years	<40 Years	40–49 Years	>50 Years
Low	1	0–48	0–68	0–61	0–48	0–57	0–54
	10	49–69	69–71	62–67	49–52	58–73	55–59
	20	70–73	72–78	68–75	53–64	74–79	60.-70
Medium/ Low	30	74–76	79–83	76–80	65–74	80–85	71–75
	40	77–80	84–85	81–86	75–77	86–89	76–82
Medium	50	81–85	86–93	87–89	78–82	90–94	83–85
Medium/High	60	86–93	94–97	90–92	83–93	95–96	86–88
	70	94–99	98–99	93–98	94–95	97–98	89–95
High	80	100–103	100–102	99–100	96–98	99–104	96–101
	90	104–107	103–107	101–107	99–107	105–107	102–107
	99	108	108	108	108	108	108
N = 118		16	32	32	10	14	14

,

Table 7. Gender and service-learning experience-based scale.

Level	Percentile	Female		Male
		Beginner (<5 Years)	Experienced (>5 Years)	Beginner (<5 Years)	Experienced (>5 Years)
Low	1	0–48	0–61	0–48	0–54
	10	49–68	62–75	49–55	55–74
	20	69–70	76–80	56–57	75–77
Medium/Low	30	71–73	81–85	58–66	78–83
	40	74–78	86–88	67–73	84–85
Medium	50	79–82	89–93	74–82	86–88
Medium/High	60	83–89	94–97	83–93	89–93
	70	90–98	98–99	94–95	94–97
High	80	99–100	100–102	96–100	98–99
	90	101–107	103–107	101–107	100–107
	99	108	108	108	108
N = 118		31	49	14	24

,

Table 8. Scale based on gender and type of institution in which service-learning is practiced.

Level	Percentile	Female		Male
		Social, Community, and Health Intervention	Educational Intervention	Social, Community, and Health Intervention	Educational Intervention
Low	1	0–65	0–48	0–54	0–48
	10	66–70	49–68	55–61	49–63
	20	71–77	69–74	62–73	64–72
Medium/ Low	30	78–80	75–79	74–78	73–76
	40	81–85	80–84	79–84	77–83
Medium	50	86–92	85–88	85–88	84–93
Medium/High	60	93–97	89–93	89	94
	70	98–99	94–97	90–95	95–96
High	80	100–102	98–100	96–98	97–101
	90	103–107	101–107	99–107	102–107
	99	108	108	108	108
N = 118		38	42	22	16

and

Table 9. Scale based on gender and nature of the service-learning experience.

Level	Percentile	Female		Male
		SL Is Exclusively Face-to-Face	SL Includes Totally or Partially Online Activity	SL Is Exclusively Face-to-Face	SL Includes Totally or Partially Online Activity
Low	1	0–48	0–61	0–54	0–48
	10	49–70	62–68	55–68	49–54
	20	71–75	69–76	69–75	55–64
Medium/Low	30	76–78	77–82	76–77	65–77
	40	79–83	83–88	78–84	78–84
Medium	50	84–87	89–91	85–88	85–93
Medium/High	60	88–94	92–97	89–92	94–95
	70	95–97	98–99	93–95	96–97
High	80	98–102	100	96–101	98–99
	90	103–107	101–107	102–107	100–107
	99	108	108	108	108
N = 118		48	32	28	10

). These scales have been calculated considering the mean and standard deviation of each item based on the variables defined above.

6. Discussion

QaSLu-27 stands out as one of the most concise questionnaires currently available. It has fewer items than comparable instruments such as those developed by Rodríguez-Izquierdo [2], Escofet-Roig et al. [20], Ruiz-Ordoñez et al. [23], and Gul et al. [15], which contain 8, 16, 20, and 21 items, respectively. While some validated instruments feature even fewer items, such as the 30-item questionnaire by Rodríguez-Izquierdo [21], the 35-item version by León-Carrascosa et al. [9], and the 41-item questionnaire with an open question by Santos-Pastor et al. [22], QaSLu-27 strikes a balance by encompassing 27 items.

It is noteworthy that QaSLu-27 underwent a Delphi method validation, similar to other instruments like those developed by Escofet-Roig et al. [20], León-Carrascosa et al. [9], López-de-Arana et al. [24], and Rodríguez-Izquierdo [21]. However, following the processes of different authors [2,9,21,22,23], QaSLu-27 has also been exposed to a factorial analysis. Although the tendency of the authors when validating the questionnaires has been to carry out a CFA, this study has sought to optimize QaSLu-27 as much as possible, ensuring maximum validity, robustness, and reliability. This methodology aligns with the meticulous work of Rodríguez-Izquierdo [2,21].

In terms of reliability, the small number of items in QaSLu-27 did not compromise its reliability. Knowing that Cronbach’s alpha is especially sensitive to this issue [34], QaSLu-27 has a notably high Cronbach’s alpha coefficient of 0.92. When contrasting the reliability of QaSLu-27 with other instruments, it can be observed that the tool developed by Rodríguez-Izquierdo [21], despite having more items, has slightly lower reliability (0.87); and the tool validated by Rodríguez-Izquierdo [2], with fewer items, has much lower reliability (0.70). The instruments developed by Escofet-Roig et al. [20] and Ruiz-Ordoñez et al. [23], despite having fewer items, are around 0.90 and 0.67, respectively. However, the proposals of León-Carrascosa et al. [9] and Santos-Pastor et al. [22] achieve a higher reliability (0.95), but it should be noted that they have more items. Only the instrument by Gul et al. [15] surpassed QaSLu-27 with a reliability of 0.96, attributed to the significantly larger sample size in their study.

This study provides a questionnaire, the QaSLu-27, which is shorter and more reliable than many of the existing instruments. Therefore, the reduction in the length of the QaSLu-27 questionnaire does not compromise its quality. In addition, this work introduces some scales that could serve as a starting point from which to continue feeding those already constructed and/or to continue constructing new scales that help evaluate and contrast service-learning experiences. Those scales allow educators to evaluate their service-learning experience by taking into account specific characteristics, a feature that was not available until now, and they also allow educators to improve their service-learning practice. Finally, the main limitation is the sample, as it is restricted to experienced educators from diverse disciplines and universities. While other studies may have larger samples, they may lack the expertise of this sample because they survey students, and they may also lack diversity (different types of higher education institutions and different disciplines).

7. Conclusions

The proliferation of service-learning experiences in various disciplines in higher education makes it necessary to develop instruments that ensure their quality and reliability.

This study addresses the deficiencies identified in the theoretical framework by providing a valid, robust, reliable alternative.

First of all, it should be noted that the psychometric validation sample comprised service-learning expert educators from diverse disciplines and institutes of higher education, ensuring a comprehensive assessment.

Also, QaSLu-27 boasts a robust design, adhering to the principle of parsimony with a single-factor configuration.

Moreover, it reflects a highly discriminating total score of the quality of service-learning experiences, which is a requirement for developing comparative scales.

Importantly, this questionnaire not only aids in assessing service-learning experiences but also offers support in their design by providing information to improve it.

Finally, while QaSLu-27 has been validated with participants from universities in Spain, it would be interesting to expand its use by involving educators from institutions of higher education in other European countries to achieve a new validation and evaluation.