Self-Assessment Guide to Quality in Accessible Virtual Education: An Expert Validation

Abstract

The evaluation of quality in education has gained greater relevance, especially after the COVID-19 pandemic, which enhanced virtual learning and further highlighted the need for inclusive, equitable, and quality education in line with SDG 4. It has become indispensable to have evaluation models and tools supported by empirical validation to ensure their applicability and reliability in diverse educational contexts. This study addresses this need by validating a self-assessment guide designed to evaluate the quality of virtual education from an accessibility perspective. The validation process involved three phases: a preliminary phase with 4 experts to identify areas for improvement in the structure, clarity, and relevance of the content; an extended phase with 20 experts to evaluate reliability and validity; and a pilot study implemented in four higher-education institutions to verify the guide’s applicability in real environments. Reliability and validity were assessed using standard psychometric techniques, confirming high internal consistency and content validity, although some areas required clarity adjustments. The findings indicate that this guide can serve as a robust tool for institutions aiming to advance accessible and high-quality virtual education, directly supporting SDG 4’s commitment to inclusive education.

1. Introduction

The pursuit of sustainability is now a top priority for educational institutions worldwide, serving as a fundamental pillar to ensure not only compliance with current standards but also preparedness to face future challenges [1,2]. In this regard, institutions are constantly confronted with the need to guarantee the quality of their programmes and services, as well as their long-term sustainability, with the aim of equipping students with the competencies required by the labour market [3]. Institutions are thus tasked with not only upholding the quality and sustainability of their programmes but also aligning their educational outcomes with the competencies demanded by the labour market, a need underscored by Sustainable Development Goal 4 (SDG 4) [4] in the United Nations’ 2030 Agenda [5], which aims to ensure inclusive, equitable, and quality education for all.

In this context, the quality of higher education has gained unprecedented relevance, especially after the COVID-19 pandemic, which spurred a rapid transition to virtual and hybrid learning environments [6,7]. However, this shift revealed significant challenges regarding the quality and accessibility of such environments [8,9]. Accessibility not only ensures that students with disabilities can access content but also impacts aspects such as usability, intuitive navigation, and the inclusion of all student profiles [8], making it a key element of quality in virtual education [10].

Although there are multiple standards or models designed to assess the quality of virtual or distance education [11,12,13], many of them lack solid empirical validation, maintaining a predominantly conceptual nature [14,15,16]. This lack of validation compromises their large-scale practical applicability and their capacity to generate precise and applicable diagnostic results in various institutional contexts [17]. Furthermore, these models lack the necessary reliability to support strategic decisions based on their results.

This study addresses this gap by setting forth the primary objective of empirically validating a self-assessment guide for quality in virtual education. The purpose is to provide a practical and reliable tool that enables higher education institutions to evaluate and improve their virtual education processes, aligning with the principles of SDG 4 and promoting a more inclusive and sustainable education. In this sense, the research question that guided the study was as follows: Is it possible to consolidate and validate a self-assessment guide for quality in virtual education from an accessibility perspective that is applicable across different institutional contexts? Two sub-research questions arise from this approach:

• Are the structural elements of the proposed model sufficient and do they effectively facilitate its application in accessible and equitable virtual environments in higher education?
• Are the scientific basis and content of the proposal sufficiently robust to support its practical application and adaptation in the context of accessible and equitable virtual higher education?

To answer these questions, an empirical validation of the reliability and validity of our self-assessment guide was conducted [18], using both quantitative and qualitative analyses. The validation was based on expert judgement [19] and, subsequently, a pilot study. Through an iterative validation process, an improved version of the guide was developed, ensuring that it meets the standards and expectations of experts in the areas of accessibility and educational accreditation. Different statistical techniques were applied in the process, allowing for rigorous internal consistency, content validity, and the guide’s applicability in various institutional contexts.

The findings of this article provide empirical evidence on the effectiveness of the proposed model, positioning the self-assessment guide as an important tool to promote quality and accessibility in virtual education.

This article is structured as follows: Section 2 presents a review of the relevant literature, including a summary of the characteristics of the self-assessment guide to be validated. Section 3 describes the methodology employed, including both quantitative and qualitative analyses, and the techniques used in the two validation phases and in the pilot study. Next, Section 4 presents the results obtained in each phase of the study. Finally, Section 5 discusses the study’s findings and Section 6 presents some conclusions.

2. Background

Given the growing focus on quality assurance in virtual education and the orientation of this research, it is essential to examine concepts that link the quality of virtual education and the current state of validation of related models, standards, and/or frameworks, which constitute the foundation of this study. This literature review is structured around four main themes: quality in virtual education, accessibility as a key element of quality, empirical methodologies in educational evaluation, and, finally, a summary of the self-assessment guide that is the object of validation in this research.

2.1. Quality in Virtual Education

The quality of virtual education has been a subject of study in recent decades, given the growing importance of this teaching–learning modality, which refers to the effectiveness and efficiency of the educational process supported by digital environments [20,21,22]. Some studies propose models for evaluating the quality of virtual education [23,24,25,26]. Similarly, other research suggests quality assessment methodologies for service based on Six Sigma and multivariate capacity analysis, aiming to improve student satisfaction and the perception of quality [21].

Other studies also highlight significant differences between the quality of virtual education and traditional education [25,27]. These differences are observed in specific dimensions and subdimensions, in the components of virtual classrooms (such as experiential learning and communication), and in support and human behaviour factors, which impact the perception of service quality [23,28]. Similarly, research is often limited and does not always take into account the diverse profiles of students, who, to succeed in virtual environments, require specific skills, such as effective time management and advanced digital competencies [29,30].

Regarding best practices to ensure quality in virtual education, notable didactic strategies include online-learning-oriented approaches, such as motivation, effective communication, innovation, and collaborative work [31]. Additionally, there is an emphasis on the importance of having trained teaching staff proficient in digital technologies and appropriate methodologies for virtual teaching, in order to ensure a quality educational experience for students [32,33].

In this broader perspective, the quality of virtual education transcends the teaching–learning process to include accessibility, usability, and the cultivation of specific skills that equip students to navigate the demands of the labour market, particularly in the post-pandemic era, in which adaptability and digital proficiency have become paramount [3].

2.2. Accessibility as a Key Element of Quality

Accessibility is a fundamental element to ensure inclusion and quality in virtual education. Accessibility refers not only to physical access to digital environments but also to usability, intuitive navigation, and adaptability to different needs [34,35]. Although there are widely recognised web accessibility standards and guidelines, such as the Web Content Accessibility Guidelines (WCAG), their application in the virtual educational context is not always straightforward [36,37]. Despite the opportunities provided by information and communication technologies (ICT) to reduce these barriers, challenges still persist for students in virtual learning, particularly for those with special needs [38].

Key challenges include the following: (a) the integration of inclusive pedagogical practices with adequate technological resources to create accessible learning environments, emphasizing the importance of personalised teaching/learning services and Universal Design for Learning [39,40]; (b) the need for a systematic approach to improving accessibility that addresses the lack of interest and awareness about accessibility in education [41,42,43].

Despite efforts made in previous studies to address the need for quality in higher education, many of these do not consider accessibility as a central aspect. In response to this, the authors of [43] propose a comprehensive framework to enhance accessibility in post-pandemic higher education, with the potential to become a global reference for institutions interested in implementing continuous improvement processes in virtual or hybrid programmes. As part of this proposal, an accessible e-learning quality self-assessment guide was developed, which includes a set of dimensions and criteria for institutions to evaluate the quality of their virtual programmes, taking accessibility into account [18].

2.3. Models and Empirical Methodologies in Educational Evaluation

As mentioned earlier, although there are models that emphasise quality in virtual education [11,12,13], many lack rigorous empirical validation, supported by valid statistical methodologies and techniques [13,14,15], Additionally, while other studies reflect empirical validation, they do not provide details on the psychometric measures applied [44,45,46,47], compromising their applicability and the validity of potential results if implemented in real-world contexts [17].

On the other hand, the systematic review of the literature on the standardised assessment of learning highlights the importance of addressing statistical properties [48]. For example, the calculation of Cronbach’s Alpha, widely used in educational research, is key to evaluating the reliability of instruments and ensuring the internal consistency of scales used in learning assessments [49]. Similarly, another study describes the process of designing and constructing an instrument to assess mathematical competence, emphasising the importance of expert judgment and the calculation of the Content Validity Index (CVI) to ensure the quality of the instrument [50]. Furthermore, a study following a similar procedure mentions the need for a future pilot test to analyse the reliability and internal consistency of the instrument [51].

Other studies [51,52,53,54], to enrich the validation process, consider expert judgement and pilot testing with users, integrating psychometric measures to strengthen the robustness of the results. For example, Licen Sabina [52], in her study on the creation and validation of tools for assessing the quality of e-learning courses, in addition to carrying out content validity analysis with the CVI, carries out a confirmatory factor analysis for construct validity and Cronbach’s Alpha coefficient to assess reliability. Hernández and Juárez [51], in their study on the validation of an instrument to evaluate tutorial action in higher education, complemented content validity with the calculation of confidence intervals (CVI). Farid [54], on the other hand, applied several statistical tests, such as principal component analysis, logistic regression, the chi-squared test, and mean analysis to analyse empirical data.

In this regard, having evaluation tools with adequate statistical properties is crucial to ensure the reliability and validity of results [11,55]. The statistical properties of evaluation tools allow, for example, the identification of the extent to which the instrument truly measures what it is intended to assess, as well as the consistency of the scores obtained in an assessment [48], implying that the measurements are reliable over time. This is particularly relevant in the design and construction of standardised proposals, in which the rigour and quality of the instrument are essential to ensure its reliability and validity.

2.4. Self-Assessment Guide

The self-assessment guide [18] was proposed as a response to the need for quality assurance in virtual programmes within educational institutions from the perspective of accessibility and inclusion.

Despite numerous efforts to improve virtual learning environments, such as the incorporation of emerging technologies in the teaching–learning process [56,57,58] and the development of models and tools focused on quality, an integrated approach that places accessibility at the core of this quality is still lacking [11,59,60]. Current studies often isolate the concepts of quality and accessibility, focusing on individual aspects such as pedagogy, technology, management, or organisation [24,41,60,61].

In this regard, the proposed guide [18] seeks to integrate the various elements related to quality in virtual education. It establishes dimensions and criteria that enable institutions to conduct a comprehensive self-assessment of their programmes, considering accessibility as a transversal axis. For this purpose, a metamodel has been constructed based on an exhaustive analysis of existing models, standards, and proposals on quality in both face-to-face and virtual education, as well as the challenges of inclusion and accessibility in virtual education. This metamodel takes a comprehensive approach to educational quality and is characterised by its adaptability to diverse institutional contexts.

One study [18] proposes a multidimensional approach to ensuring the quality of e-learning, addressing accessibility from the perspectives of institutional organisation, the student body, teaching, and the technological infrastructures of institutions.

The following briefly describes the phases followed for the development of the guide (see more details in [18]).

• Systematic literature review (SLR): An exhaustive review of the literature was conducted using the Multivocal Literature Review (MLR) methodology [62] to identify models and criteria for quality assessment in e-learning. Additionally, it included procedures for self-assessment, evaluation, and/or accreditation in contexts of quality assurance in education.
• Identification of dimensions and criteria: Based on the SLR results, 134 dimensions and 504 criteria related to accessibility were identified. A thematic analysis was then conducted, resulting in 18 grouped dimensions and 40 phases or sub-processes applicable to self-assessment.
• Model construction: Based on the dimensions and criteria identified, focus groups were conducted with a panel of experts in accessibility and educational accreditation. A model was proposed consisting of four main dimensions (Organisation, Student Body, Teaching, and Infrastructure), with 16 standards, 48 requirements, and 63 pieces of evidence, as shown in Figure 1.
• Construction of the self-assessment methodology: From the identified phases, a methodology was proposed for implementing the model. Figure 2 shows the proposed model, which includes five phases (Planning, Model Tuning, Evaluation, Results, and Continuous Improvement), with 24 clearly defined activities.
• Design of the guide and preparation of the final document: The guide was structured into two key components: (a) the self-assessment model and (b) the self-assessment methodology. The final guide includes a conceptual framework, the self-assessment model, and a detailed methodology for its implementation.

3. Materials and Methods

The present study is structured by a methodological research design [63]. Therefore, an interactive validation procedure for the self-assessment guide on accessible virtual education is reviewed and proposed, comprising the following three phases:

(a)

Preliminary validation by a small group of high-level experts with the aim of identifying critical areas for improvement in the structure, clarity, and relevance of the guide’s content.

(b)

Expanded validation based on the improved version of the guide, with the aim of confirming and further exploring the internal consistency and criterion concordance among experts. The result is detailed feedback for making further improvements to the guide based on a larger and more diverse group of experts.

(c)

Implementation validation through a pilot study of the final version of the guide in four educational institutions. The objective was to identify the guide’s adaptability to real institutional environments, as well as any necessary adjustments for its implementation in a broader range of educational institutions.

Quantitative data analysis was conducted using RStudio 2024.04 and Jamovi 2.3 software [64,65], through which statistical analyses were generated to address the study’s objectives. Microsoft Excel 365 was used for the qualitative analysis.

All the instruments used in this study, including the self-assessment guide and the expert assessment questionnaires, among others, were written and applied in Spanish, the primary language of the participants. Although two of the experts were native Portuguese speakers, they were proficient in Spanish, so no linguistic adaptations were required. For the purposes of this article, some extracts from the instruments have been translated into English to facilitate their comprehension.

Each phase of the study was carefully designed to ensure that the guidance was not only theoretically sound, but also applicable and effective in real-world contexts, contributing to the continuous improvement of participating institutions.

3.1. Experts Involved

The preliminary validation, as well as the extended validation of the guideline, involved the participation of carefully selected experts to ensure relevance and depth of analysis. Inclusion criteria were defined for each validation. In the preliminary validation, a panel of experts was considered, whose background included the following:

• Proven experience in quality evaluation of educational programmes.
• Knowledge and skills in web accessibility and universal design for learning to ensure that the guide adequately addresses these aspects.
• Publications or participation in research projects related to accessibility and/or quality evaluation in education.

The extended validation considered the following:

• Proven experience in the areas of virtual education.
• Proven experience in quality evaluation for educational programmes and/or knowledge and skills in web accessibility and universal design for learning to ensure that the guide adequately addresses these aspects.

Table 1. Profile of experts who participated in the study.

Variable	Category	Validation Initial	Validation Extended
Gender	Female	1	10
	Male	3	10
Age	31–39 years		5
	40–49 years	1	8
	50–59 years	3	7
Country	Cuba		1
	Ecuador	1	12
	Spain	2	2
	Mexico		3
	Portugal	1	2
Highest level of education	Magister	1	7
	PhD	3	13
Profession	University lecturer	4	20
	Another	0	0
Area in which you work	Social		1
	Education	1	4
	Engineering	1	3
	Computing	2	9
	Administration		2
	Health		1
Experience in educational quality evaluation	No	0	2
	Yes	4	18
Role in the evaluation process	Assistant evaluator		1
	Internal evaluator		2
	Internal and external evaluator	3	11
	External evaluator	1	1
	Coordination	1	3
Years of accreditation and assessment experience	0 years		2
	1–5 years		8
	6–10 years		6
	11–15 years	1	3
	16–20 years		0
	21–25 years	2
	26–30 years	1	1
Virtual or LMS-mediated training time	1–5 years		6
	6–10 years		9
	11–15 years	1	2
	16–20 years		2
	21–25 years	3	3
Years of experience in accessibility	0 years		4
	1–5 years		6
	6–10 years		5
	11–15 years		3
	16–20 years	4	1

presents the profile of the experts who participated in the study. The profile of the preliminary validation experts was characterised by a homogeneous group of university professionals, with an average age of 53.8 years and considerable experience in accreditation and evaluation, averaging 20.8 years. Additionally, they had strong academic and research backgrounds, with a specialised focus on accessibility.

On the other hand, the experts in the expanded validation presented a more diverse profile in terms of gender, with an equal representation of men and women. The mean age of this group is 45.9 years, suggesting a combination of younger experts and those with significant career trajectories. The participants had solid academic backgrounds, with the majority holding a PhD. Furthermore, this group worked in a variety of fields, including technology, engineering, education, and life sciences, among others. Their experience in accreditation and evaluation was reduced compared to the initial group, with an average of 7.10 years, but they had significant training in accessibility and in the use of virtual learning platforms, offering another perspective to the evaluation.

3.2. Preliminary Validation

The focus of this first phase of the research was to obtain quantitative and qualitative feedback on the coherence and relevance of the guide, to make necessary adjustments based on the experts’ assessments. The objectives of this phase were (O1) to assess the content validity of the guide and (O2) to describe the experts’ observations and apply them as adjustments to improve the guide.

To fulfil the first objective (O1), Aiken’s V methodology [66] was used, a methodology that allows us to quantify the degree of agreement among experts on the relevance of each item in relation to the construct to be measured. This technique is widely used in psychometric research to ensure that the selected items adequately reflect the proposed theoretical content. In this case, Aiken’s V index was calculated for each item evaluated, and then averages of the index were obtained for each of the aspects evaluated (practical value, content, structure, configuration of the guide).

To fulfil the second objective (O2), a thematic qualitative analysis was carried out [67] based on the experts’ responses to the questions asked. The thematic analysis identified recurring patterns and categories, revealing both strengths and areas in need of adjustment, especially in terms of internal coherence and practicality of content.

The first validation or expert review was carried out based on an evaluation rubric (questionnaire) consisting of 27 questions, which included 4 important aspects of the guide: practical value, guide content, guide structure, and guide configuration. These aspects were evaluated using a 5-point Likert scale, along with open-ended questions to obtain a broad perspective on the structure, content, and applicability of the guide (see details in Supplementary Materials File S1).

The survey was initiated by selecting and inviting experts via email to participate in the validation. A total of six invitations were sent to experts in the areas of evaluation, accreditation, and accessibility. These experts were selected due to their extensive experience in these areas and because of a previous collaborative relationship in shared research work and projects with the authors. From the six invitations sent, four experts confirmed their participation. Once the experts confirmed their participation, the guide was provided to them, along with a set of detailed instructions on their participation and how to complete the associated questionnaire. To facilitate participation and ensure efficient data collection, the questionnaire was made available electronically through a Google Forms form.

As a result of this first phase, adjustments were made to the guide regarding the clarity and relevance of the proposed content before extending the process to a larger group.

3.3. Extended Validation

The focus of this second phase was to verify whether the changes made in the first phase were relevant and whether improvements were made in the understanding and application of the guide. Additionally, this phase aimed to refine the guide to ensure that it was not only theoretically robust but also functional, applicable in practice, and adaptable to the needs of each institution. The objectives pursued in this phase were as follows:

(O1) To analyse the internal consistency of the validation instrument. The dimensions of the self-assessment model and the phases of the guide’s methodology were evaluated separately to determine the consistency of the items within each category. The coefficients used were Cronbach’s Alpha [68] and McDonald’s Omega [69]. These are widely recognised in the psychometric literature as indicators of an instrument’s internal reliability.

(O2) To validate the criterion agreement among experts in the validation instrument. The aim was to measure the agreement between experts when making an evaluation, i.e., how similar the responses are when they assess the study instrument. In this case, the intraclass correlation coefficient (ICC) [70] was used. The ICC is a statistic that evaluates the reliability of measurements made by different observers, measuring the consistency or degree of agreement among them across a set of items. For the calculation, the average measures were used, averaging the evaluators’ scores for each item, to obtain a more general evaluation, less influenced by individual variability among evaluators.

(O3) To reassess the content validity of the guide using an expanded panel of experts. The content validity of the guide was reassessed, evaluating both the self-assessment model (dimensions) and the methodology (phases). A similar procedure to the preliminary validation was applied. A quantitative analysis was conducted using Aiken’s V index for each evaluated item, complemented by a qualitative analysis based on a thematic analysis following the phases established [67]. For the thematic analysis, after a thorough review of all observations, each observation was segmented and coded around the four evaluated criteria (sufficiency, coherence, relevance, and clarity). After coding/categorising the observations, refinement and identification of patterns in the responses were carried out, supported by the association with keywords. Based on these patterns, necessary adjustments and improvements to the guide were identified. Additionally, a criterion was applied to accept or reject suggestions, based on the results of Aiken’s V index. If Aiken’s V showed a high value, it justified maintaining the item without changes. In cases in which the values were lower, the consistency and coherence of the suggestion patterns were prioritised.

(O4) To compare the expanded expert evaluation of the dimensions, phases, and criteria of the guide’s evaluation. Finally, to ensure the robustness of the guide, the aim was to identify patterns of consistency or discrepancy in the evaluators’ responses, which would indicate which dimensions or phases showed variability or critical areas requiring review or adjustment. A comparison of the scores obtained by the dimensions, phases, and criteria used in the expanded evaluation of the guide was conducted.

The comparative analysis followed the following procedure: (a) the mean and median were calculated to represent the central behaviour of the scores assigned by the experts; additionally, the standard deviation was calculated to analyse the dispersion of the responses, which allowed the identification of the degree of variability in the scores; (b) the Shapiro–Wilk test was applied to assess the normality of the data and thus deepen the interpretation of the results; (c) diagrams were constructed to graphically display the distribution, using boxplots due to the non-normal distribution of the responses. The interpretation or reading of the boxplots focused on the position of the median (line within the box), the interquartile ranges (limits of the box containing the central 50% of the scores), and the outliers (points outside the box range) [71]; (d) a comparison of the interquartile ranges between dimensions, phases, and criteria was conducted, paying particular attention to variables whose ranges showed little overlap, which allowed for the detection of possible significant differences between the experts’ responses, suggesting items to review and adjust; (e) patterns of variability in the responses were identified, particularly those with the highest standard deviation, which allowed for the identification of items that required a more detailed review, with the aim of improving their clarity and relevance.

The expanded validation was based on the administration of a second evaluation questionnaire to a group of 20 experts (see details in Supplementary Materials File S2). The questionnaire was designed to evaluate four fundamental criteria of sufficiency, coherence, relevance, and clarity, as detailed in

Table 2. Evaluation criteria for the self-assessment guide.

Criterion	Description	Rating Scale
Sufficiency	The items (standard/activity) that belong to the same dimension/phase are sufficient to measure it.	1–4 a
Clarity	The item (standard/activity) is easily understood, i.e., its syntax and semantics are adequate.	1–4 a
Coherence	The item (standard/activity) is logically related to the dimension/phase or indicator it is measuring.	1–4 a
Relevance	Does the self-assessment guide allow institutions to improve their educational service?	1–4 a

^a 1 = does not meet the criterion, 2 = low level, 3 = moderate level, 4 = high level.

. It was structured with a dual purpose:

(1)

To validate the proposed self-assessment model by asking the evaluators to rate the sufficiency of each of the four dimensions and the coherence, relevance, and clarity of each of the standards (16 in total), with the option to provide constructive feedback for each standard.

(2)

To validate the proposed self-assessment methodology, in which experts were asked to assess the sufficiency of each of the five methodological phases and evaluate the coherence, relevance, and clarity of each activity within a phase (24 in total). For each activity, experts were also given the opportunity to provide constructive feedback.

Although each evaluator had to answer up to 165 questions (126 quantitative and 39 qualitative), the following measures were taken to mitigate the risk of “survey fatigue”: (a) flexibility in response deadlines, (b) clear instructions regarding the participant’s procedure, to be carried out in at least 3 sessions, with the first session to study the guide, the second to answer the questionnaire related to the self-assessment model, and, finally, the third to answer the questionnaire related to the proposed self-assessment methodology, (c) agility in completing the questionnaire by using an interactive form to facilitate its completion and minimise errors, such as incorrect or out-of-context responses, and d) the establishment of a communication channel for queries from the experts regarding the guide.

The questionnaire procedure began with the selection and invitation of experts to participate in the validation. The selection of experts was made from a database that included members of educational evaluation and accreditation commissions from one of the authors’ universities, as well as researchers specialising in educational evaluation and accreditation. In addition, researchers with experience in projects related to the topic with whom a previous collaborative relationship existed were included.

Invitations were sent to a total of 30 experts. Of these, 23 confirmed their participation, and complete responses were finally received from 20 experts, representing a response rate of 87%. After confirming their participation, the guide was sent along with a set of detailed instructions on how to participate in the validation and how to complete the questionnaire. These instructions were complemented by an explanatory video that described the objectives of the evaluation and detailed the participation procedure (see details in Supplementary Materials Video S1).

During the validation process, continuous and proactive communication was maintained with the experts via email. This exchange allowed questions to be addressed and additional impressions of the guide and the validation process to be collected, with the aim of improving the quality and relevance of the responses obtained.

As a result of this second round, improvements were made to the guide in relation to the criteria of sufficiency, coherence, relevance, and clarity. The improved version of the guide was implemented in real contexts to assess its applicability.

3.4. Pilot Study

After completing the validation process for the two previous phases, a pilot study of the self-assessment model for the quality of virtual education was carried out in four higher-education institutions, two of which were located in Ecuador and two in Mexico.

Based on the final version of the guide, a tool was developed using a spreadsheet. This tool was specifically designed to facilitate the implementation of the guide and capture all the critical aspects of a self-assessment (see details in Supplementary Materials Tool S1). The tool includes the following parts:

(a)

General information: For the registration of basic data from the participating institution, such as institution name, level of education (undergraduate, postgraduate), scope of application (institutional, degree or programme, course), mode of study (online, hybrid), date or period of evaluation, and names of the evaluator(s) and informant(s).

(b)

Self-assessment model: This section details the model, structured by dimensions, with a description and objective for each dimension and its corresponding standards. For each standard, the name, objective, rating scale, and each of the standard’s requirements are included. For each requirement, a description, evidence or sources of information, and examples of aspects that could justify the requirement’s fulfilment are provided. These examples serve as guidelines for determining the rating at the time of evaluation.

(c)

Rating for each standard: This section provides a detailed description of each standard, along with its corresponding requirements. For each requirement, a weight or score is assigned according to institutional relevance (critical, important, or desirable), determining applicability, and recording the fulfilment rating. A list of possible evidence to justify the fulfilment of the standard is also included, with checkboxes to indicate whether the evidence exists and whether it effectively contributes to meeting the requirement, along with a space for comments on the characteristics of the evidence. Additionally, the tool automatically calculates the final rating for the standard, taking into account the assigned weight and the scores obtained for each requirement. A qualitative rating and a field to describe the overall perception of the standard within the institutional context are also included, along with a space for proposing improvements for each evaluated standard.

(d)

Weighting for scoring: This section contains a summary of the weightings assigned by the institution (evaluator and informants) to each requirement and standard, allowing the weightings of the standards and requirements to be adjusted, ensuring that the guide is adaptable to the specific characteristics of each institution.

(e)

Results: This section provides a summary of the scores obtained and their equivalence in terms of meeting or not meeting the standard. It is complemented by “radar” charts, which provide a clear visualisation of the institution’s status in relation to the different evaluated dimensions, thus facilitating the interpretation of the results and the identification of priority areas for continuous improvement.

The pilot was carried out following seven stages: (1) initial session to present and formalise the self-assessment process, including the presentation of the model, its objectives, and the relevance of the process for improving the quality of virtual education, (2) training on the use of the tool, with an emphasis on interpreting the standards, the weighting methodology, and the rating procedure for each standard, (3) distribution of forms to the self-assessment team, (4) implementation of the self-assessment using the designed forms, (5) support throughout the process to resolve questions and provide immediate feedback to ensure the correct application of the model, (6) analysis of the results obtained and preparation of the final report, and (7) collection of feedback on the functionality of the forms and the effectiveness of the self-assessment model.

The purpose of this phase was to assess the applicability of the model in real institutional contexts, observing how it adapts to the diverse realities of the participating educational institutions. Additionally, this phase aimed to identify both the strengths and critical areas of the model to enhance quality in terms of accessibility and inclusion in virtual education. The objectives established for this phase were as follows:

(O1) To evaluate the model’s adaptability to different institutional contexts. Participating institutions were asked to apply the model, weighting each standard and requirement according to the relevance they deemed appropriate for their own context. This approach allowed for differentiated weighting of the standards, adapting the model to the specific characteristics of each institution.

(O2) To validate the model’s applicability in real environments. The participating institutions conducted a self-assessment in the four dimensions, completing the questionnaire according to the model’s standards. This process enabled an observation of how each institution perceives the model and gathered information on its functionality and applicability under real self-assessment conditions.

4. Results

This section presents the results obtained from the validation process and the pilot study of the self-assessment guide. The findings from each stage are detailed below.

4.1. Preliminary Validation

As a result of this validation, necessary adjustments were identified to improve the clarity and coherence of the guide’s content. Regarding the quantitative analysis,

Table 3. Results of the content validity analysis using Aiken’s V index.

Aspect Assessed	Number of Questions	Aiken’s V a
Practical value of the guide	5	0.86
Contents of the guide	2	0.81
Structure of the guide	17	0.88
Guide configuration	3	0.94
Total	27	0.88

^a. A value close to 1 indicates a high level of agreement, values above 0.70 indicate adequate agreement, while lower values suggest a need for revision or that it does not adequately reflect the construct.

shows the average assessment of content validity using Aiken’s V methodology, which highlights a high level of agreement between the evaluators on the four assessed aspects of the self-assessment guide (see details in Supplementary Materials Table S1).

This analysis aimed to achieve objectives 1 and 2. As a result of this validation, necessary adjustments were identified to improve the clarity and coherence of the guide’s content. Regarding the quantitative analysis, Table 3 shows the average content validity evaluation using Aiken’s V methodology, which highlights a high level of agreement among the evaluators regarding the four aspects evaluated in the self-assessment guide (see details in Supplementary Materials Table S1)

The practical value of the guide achieved an average Aiken’s V of 0.86, indicating that the experts consider it a valuable tool for evaluating the quality of accessible virtual education. This perception of the guide as a practical resource highlights its relevance in educational evaluation in virtual environments, underscoring its potential to improve the educational services offered by institutions that implement online training programmes.

Regarding the content of the guide, as a result of objective (O1), the average Aiken’s V index was 0.81, reflecting general acceptance, albeit with some minor discrepancies in the ease of understanding of the contents. This revealed the need to make minor adjustments in the wording or presentation of the content to ensure greater clarity and uniform understanding.

While the “structure of the guide” aspect received a solid rating, with an Aiken’s V of 0.88, highlighting the coherence and sufficiency of the proposed standards and phases, a more detailed analysis revealed that the question, “Do you agree with the weighting and scoring proposed in the self-assessment guide for each of the dimensions and their standards?” received a low Aiken’s V (0.56). This result indicates that the evaluators had differing opinions on the appropriateness of the weighting and scoring proposed in the guide (see Appendix A 

Table A1. Aiken’s V content validity score for a particular item.

	Eval 1	Eval 2	Eval 3	Eval 4	Aiken’s V
Question 17 on the structure of the guide
Do you agree with the weighting and scoring proposed in the self-assessment guide for each of the dimensions and their standards?	3	4	3	3	0.56

), which prompted the need to seek ways to improve the acceptance of the section of the guide related to the weightings and scoring within the self-assessment model.

As a result, in a new version of the guide, the following text was added to the section on Weighting and Scoring, suggesting that evaluators are free to adjust the weighting according to their contextual reality:

“The relative evaluation of each dimension, standard, and requirement (weighting) is suggested to be analysed and determined according to the level of importance conceived by the self-assessment team, based on the institutional reality and the level or scope of application of the evaluation model”.

Finally, the guide’s configuration received a high rating, with an Aiken’s V of 0.94, suggesting an almost unanimous consensus on the quality of the language and organisation of the content. Evaluators considered that these elements facilitate the guide’s use by institutions.

As a result of the qualitative analysis, the experts’ suggestions and observations were incorporated to improve the instrument, resulting in a new version of the guide to be used in the second phase, the extended validation. It is important to note that all four evaluators commented on aspects related to the weighting and scoring proposed for the model, which aligns with the lower Aiken’s V value previously mentioned. From the thematic analysis, two categories emerged: clarity and sufficiency. Evaluator one (Eval. 1) recommended adding an appendix with practical examples of the guide’s application to facilitate its understanding. Another (Eval. 2) suggested including terms like “MOOC” and addressing connectivity, highlighting its relevance after the pandemic. A third evaluator (Eval. 3) considered the guide comprehensive but recommended greater clarity on accessibility and inclusion, as well as a more detailed development of the continuous improvement phase. Finally, a fourth evaluator (Eval. 4) emphasised the need for clarity in the guide’s objectives and proposed that updating the guide should be considered as part of the continuous improvement process.

4.2. Extended Validation

In the extended validation, which involved a larger number of experts (n = 20), the aim was to not only assess content validity but also the internal consistency and coherence of the proposed self-assessment model.

4.2.1. Internal Consistency

As a result of objective one of the extended validation phase (O1), excellent internal consistency was obtained in most of the dimensions and phases, with Cronbach’s Alpha (α) and McDonald’s Omega (Ω) values exceeding 0.9, indicating that the items within each phase and dimension are consistent with each other. The only exception was the sufficiency of the phases, which showed acceptable reliability but will need to be reviewed in future studies to improve this aspect.

Table 4. Internal consistency of dimensions.

Dimension	Items	α	Ω	Interpretation
Organisation	16	0.902	0.904	Excellent reliability
Student Body	10-1 a	0.720	0.764	Acceptable reliability
Teaching	16	0.841	0.857	Good reliability
Infrastructure	10	0.885	0.911	Excellent reliability
General	52-1 a	0.943	0.948	Excellent reliability

^a One item was removed from this set due to a lack of variability, as it was scored with the maximum value of 4 in all cases.

,

Table 5. Internal consistency of the phases.

Phases	Items	α	Ω	Interpretation
Planning	13	0.923	0.945	Excellent reliability
Model Tuning	13	0.903	0.905	Excellent reliability
Evaluation	16	0.963	0.967	Excellent reliability
Results	19	0.909	0.919	Excellent reliability
Continuous Improvement	13-1 a	0.872	0.895	Very good reliability
General (All phases)	74-1 a	0.973	0.976	Excellent reliability

^a One item was removed from this set due to a lack of variability, as it was scored with the maximum value of 4 in all cases.

and

Table 6. Internal consistency of the four criteria.

	Dimensions			Phases			Total			Reliability
Grouping	Item	α	Ω	Item	α	Ω	Item	α	Ω
Sufficiency	4	0.742	0.767	5	0.392	0.500	9	0.660	0.663	Acceptable
Coherence	16-1 a	0.846	0.871	23	0.938	0.942	38-1 a	0.926	0.938	Excellent
Relevance	16	0.864	0.879	23-1 a	0.913	0.925	38-1 a	0.916	0.933	Excellent
Clarity	16	0.869	0.876	23	0.904	0.914	39	0.869	0.876	Very good

^a One item was removed from this set due to a lack of variability, as it was scored with the maximum value of 4 in all cases.

detail the reliability results of the measurement instrument concerning the dimensions, phases, and criteria used to verify whether the guide’s content is adequate. Generally, an Alpha or Omega value above 0.7 is considered acceptable, while values above 0.8 indicate good reliability and values above 0.9 indicate excellent reliability [72,73].

In the initial analysis of the internal consistency of the four criteria (Table 6), the reliability coefficients were low (α = 0.244, Ω = 0.379), with a negative correlation for the item “Sufficiency of Phase 5 or Continuous Improvement” with the total scale, suggesting that this item was evaluating an opposing aspect. After removing it, reliability slightly improved (α = 0.371, Ω = 0.522). However, there are important theoretical reasons not to remove it, so it was reversed. After reversing the item, the reliability coefficients increased (α = 0.392, Ω = 0.500), indicating better alignment of the item with the others. Reversing the item is considered a valid attempt to improve internal coherence; however, it reveals that this phase still requires further review with a larger amount of data to achieve optimal reliability in future studies.

4.2.2. Concordance Between Evaluators

Regarding the analysis of the consistency or agreement among the evaluations of the 20 experts (O2), the ICC) provided valuable information. When using the average measurements, a value of 0.796 was obtained, which, when rounded, practically equalled a coefficient of 0.80. This indicates a high level of consistency among the experts’ evaluations when considering the average scores. This is significant because it suggests that averaging the scores reduces variability among evaluators, reinforcing the reliability of the validation instrument.

To support the ICC, it was complemented by a statistical significance analysis using the F-test. This analysis confirmed that the high level of agreement observed by the experts was not due to chance, as confirmed by a significance level of p < 0.001 (less than 1%). This provides a solid basis for trusting the consistency of the instrument validated by the experts, supporting its use in future evaluations.

4.2.3. Content Validity

As a result of the re-evaluation of the guide’s content (O3), the validity of both the model’s dimensions and the proposed phases of the methodology is presented.

Model’s dimensions:

The Organisation dimension is shown in

Table 7. Results of the evaluation of the guide according to the dimension of Organisation.

Variable	Criterion	Media	Medium	DE	Min.	Max.	Aiken’s V
Dimension 1 Organisation	Sufficiency	3.40	3.00	0.50	3	4	0.80
Standard 1.1 Organisation	Coherence	3.55	4.00	0.51	3	4	0.85
Standard 1.1 Organisation	Relevance	3.65	4.00	0.67	2	4	0.88
Standard 1.1 Organisation	Clarity	3.50	3.50	0.51	3	4	0.83
Standard 1.2 Course or academic programme information	Coherence	3.6	4.00	0.60	2	4	0.87
Standard 1.2 Course or academic programme information	Relevance	3.65	4.00	0.67	2	4	0.88
Standard 1.2 Course or academic programme information	Clarity	3.40	3.50	0.68	2	4	0.80
Standard 1.3 Economics and technology finance	Coherence	3.65	4.00	0.49	3	4	0.88
Standard 1.3 Economics and technology finance	Relevance	3.65	4.00	0.59	2	4	0.88
Standard 1.3 Economics and technology finance	Clarity	3.35	3.00	0.67	2	4	0.78
Standard 1.4 Knowledge management	Coherence	3.45	4.00	0.76	2	4	0.82
Standard 1.4 Knowledge management	Relevance	3.45	4.00	0.83	2	4	0.82
Standard 1.4 Knowledge management	Clarity	3.30	3.50	0.80	2	4	0.77
Standard 1.5 Research and innovation	Coherence	3.40	4.00	0.75	2	4	0.80
Standard 1.5 Research and innovation	Relevance	3.60	4.00	0.68	2	4	0.87
Standard 1.5 Research and innovation	Clarity	3.40	3.50	0.68	2	4	0.80
Total		3.50	3.63	0.42	2.81	4	0.83

. The evaluated standards show a generally favourable perception, although there are differences, especially in the clarity criterion, which shows greater variability.

Standard 1.1, related to organisational coherence, obtained an Aiken’s V of 0.85, indicating a high level of consensus among the experts. Furthermore, the median of 4 and the low standard deviation reinforce the idea that most of the evaluators perceive this standard as well-structured. However, clarity, with an Aiken’s V of 0.83, shows greater dispersion, suggesting areas for improvement. On the other hand, the thematic analysis revealed that nine evaluators provided observations, five of which were related to the clarity criterion. The other three were related to sufficiency, and one of them was discarded, as it merely highlighted the value of the standard. Two experts suggested “emphasising accessibility and inclusion” within this standard (Eval. 6, Eval. 7), and two others highlighted the need to clarify terms such as regulation and organisational structure (Eval. 10, Eval. 13). Another expert mentioned that supporting processes such as “quality assurance and continuous improvement” might be redundant, as the entire model itself should contribute to these objectives (Eval. 8). Regarding sufficiency, it was suggested to complement the standard with new evidence (Eval. 9, Eval. 16, Eval. 17).

Standard 1.2, related to course or academic programme information, reflected an Aiken’s V of 0.87 in the coherence criterion, confirming a high degree of agreement among the experts. However, clarity in this standard showed an Aiken’s V of 0.80, indicating greater dispersion in the responses. The relevant observations received (8) pointed out aspects related to clarity (6), sufficiency (1), and coherence (1). Regarding clarity, the suggestions focused on improving understanding by addressing aspects related to assessment, pedagogical support, and the information to be published (Eval. 6, Eval. 7, Eval. 8, Eval. 10, Eval. 12, Eval. 17). It was also suggested to include information related to course quality (Eval. 5). In terms of coherence, it was suggested to move the standard to the Teaching dimension (Eval. 9).

Finally, Standards 1.4 and 1.5, related to knowledge management and research and innovation, showed greater variability, with Aiken’s V values of 0.82 and 0.80, respectively. The observations highlighted the need to clarify procedures and clearly differentiate the requirements and evidence to facilitate evaluation.

The results for the other dimensions, as well as for the methodology phases, were similar, with consistent patterns in terms of consensus and qualitative observations (see details in Supplementary Materials Table S2).

A general analysis of the results from the evaluation of the guide’s dimensions (see Appendix A,

Table A2. Results of the evaluation of the guide according to the dimensions of the proposed model.

							Shapiro–Wilk
Dimensions	Media	Medium	SD	Min.	Max.	Aiken’s V	W	p
Organisation	3.50	3.63	0.419	2.81	4.00	0.83	0.882	0.019
Student Body	3.79	3.90	0.216	3.20	4.00	0.93	0.797	<0.001
Teaching	3.69	3.69	0.279	3.00	4.00	0.90	0.915	0.079
Infrastructure	3.79	3.90	0.334	2.70	4.00	0.93	0.688	<0.001
Criteria
Sufficiency	3.71	3.75	0.317	3.00	4.00		0.827	0.002
Coherence	3.70	3.81	0.280	3.06	4.00		0.884	0.021
Relevance	3.76	3.84	0.303	2.88	4.00		0.808	0.001
Clarity	3.53	3.53	0.347	2.75	4.00		0.943	0.268

Note 1: No items in the extended evaluation scored below 0.70, so the weights given adequately represent their specificities. For this reason, only the averages of the items in the form of the four dimensions are presented in this table. Note 2: Low W values and p-levels of less than 0.05 (<0.001) from the Shapiro–Wilk test indicate that the data do not follow a normal distribution, suggesting that non-parametric statistical methods be considered for further complementary analysis.

) reveals high scores across most of the content validity criteria, confirming the relevance and sufficiency of the elements assessed. Overall, the validity assessment of the dimensions shows a positive Aiken’s V of 0.90, confirming their content validity.

In summary, the Student Body dimension presents even stronger results, with the Aiken’s V’s reaching up to 1.00 in some standards, indicating a complete consensus on the validity of these aspects. Qualitative recommendations, such as improving terminology or including additional aspects, are useful but do not indicate a deficiency in the validity of the guide.

In the Teaching dimension, all the standards have Aiken V’s above 0.85, confirming their validity. Although clarity in some standards, such as e-assessment, could benefit from greater specificity, this does not compromise the content validity, but highlights areas where adjustments could be made to enhance understanding.

The Infrastructure dimension is also validated, with Aiken V values ranging from 0.88 to 0.97, indicating a robust consensus on the importance and relevance of the standards assessed. The qualitative recommendations focus on improving the communication of certain technical details but, again, do not compromise the validity of the guide.

Phases of the methodology:

The analysis of the phases, summarised in Appendix A 

Table A3. Results of the evaluation grouped by the phases of the self-assessment methodology.

							Shapiro–Wilk
Dimensions	Media	Medium	SD	Min.	Max.	Aiken’s V	W	p
Planning	3.83	3.96	0.297	2.85	4.00	0.94	0.627	<0.001
Model tuning	3.78	3.96	0.307	3.00	4.00	0.93	0.766	<0.001
Evaluation	3.83	4.00	0.351	2.69	4.00	0.94	0.566	<0.001
ResultsPlann	3.76	3.87	0.287	3.00	4.00	0.92	0.814	0.001
Continuos Improvement	3.88	4.00	0.200	3.23	4.00	0.96	0.666	<0.001
Criteria
Sufficiency	3.88	4.00	0.164	3.40	4.00		0.724	<0.001
Coherence	3.81	3.94	0.284	3.04	4.00		0.716	<0.001
Relevance	3.84	3.96	0.232	3.30	4.00		0.738	<0.001
Clarity	3.77	3.87	0.261	3.09	4.00		0.806	0.001

Note 1: No items in the extended evaluation scored below 0.70, thus the weights presented adequately represent their specificities. For this reason, this table only presents the averages of the items in the form of the five phases. Note 2: Low W values and p-levels of less than 0.05 (<0.001) from the Shapiro–Wilk test indicate that the data do not follow a normal distribution, suggesting that non-parametric statistical methods be considered for further complementary analysis.

(see details in Supplementary Materials Table S2), reflected a generally positive perception, highlighting very good content validity (Aiken’s V of 0.93). In the Planning phase, the activities were validated, with Aiken’s V values ranging from 0.92 to 0.98, confirming that they are adequately structured and relevant. Qualitative observations, such as the need to further detail the formalisation of the need for self-assessment, are suggestions to enrich the content, but do not affect the established validity.

During the Model Tuning phase, most activities obtained Aiken’s V values above 0.90, validating their relevance and coherence. The “redefinition of the weighting scale” showed significant improvement in the second evaluation, with Aiken’s V values between 0.85 and 0.90, indicating that the initial issue has been resolved. The observations here are qualitative and do not reflect a lack of validity but, rather. an area where additional adjustments can be made to optimise the process.

In the Evaluation phase, activities were also validated with Aiken’s V values between 0.90 and 0.97. Observations related to distinguishing between socialisation and training highlight areas for improved communication, but they do not compromise the already confirmed validity.

In the Results and Continuous Improvement phases, the standards were well evaluated, with Aiken’s V values above 0.87, supporting their validity. Although some aspects of clarity showed some variability, such as in the “issuance of evaluative judgments”, these qualitative observations do not compromise the guide’s validity.

4.2.4. Comparison of Evaluations

From the analysis of the scores assigned by the experts in the different dimensions, phases and criteria of the guide (O4), key trends were identified to assess the consistency of the responses and areas that might require adjustment. Since several of the variables analysed showed a non-normal distribution (Shapiro–Wilk test p < 0.05), boxplots were used to visualise variability and patterns in the responses. An overall positive assessment was revealed and is presented below.

Regarding the dimensions evaluated (see Appendix A Table A2), positive trends were generally observed, although with some variations that highlight areas where the guide presents strengths and opportunities for improvement.

The Student Body and Infrastructure dimensions stand out with higher scores, reflecting high consistency and agreement among the evaluators. This shows a positive perception of these aspects by the experts, with low variability (SD = 0.216 and 0.334 respectively), suggesting that these elements are well-defined and understandable. In contrast, the Organisation dimension shows greater dispersion in the responses, indicating differences in how the experts perceive aspects related to the organisation and general strategic actions that support an institution, suggesting the need for revisions to improve coherence and clarity in this area.

Applying the criteria of Sufficiency, Coherence, Relevance, and Clarity to the dimensions, it was observed that Relevance was consistently well-rated, suggesting that the contents of the guide are considered relevant and aligned with the needs of the self-assessment. However, Clarity has a lower score and greater variability, suggesting that, although the contents are relevant, the way in which they are presented could benefit from adjustments to make them more understandable and straightforward.

These observations are reflected in Figure 3 and Figure 4, highlighting the importance of considering not only the content but also the way in which information is presented in the guide, to ensure that it is perceived as clear.

Regarding the analysis of the phases of the self-assessment methodology, the evaluation results indicate a generally positive perception among the experts regarding all the phases of the self-assessment methodology, with scores close to the maximum of 4.00 across the dimensions of Planning, Model Tuning, Evaluation, Results Plan, and Continuous Improvement, and with acceptable levels of sufficiency, coherence, relevance, and clarity across the criteria assessed (see Table A3 in Appendix A for complete details).

Within these phases, the Continuous Improvement phase stands out with the highest rating (a mean of 3.88 and a median of 4.00), suggesting that the experts consider it crucial for the effectiveness of the process. The Planning and Evaluation phases follow closely, also with a mean of 3.83. However, greater variability in responses was observed in the Evaluation phase (SD = 0.351), indicating differences in the experts’ perceptions. This may suggest differences in the implementation or interpretation of the activities related to this phase, signalling the need for adjustments to ensure greater uniformity.

Regarding the criteria applied to assess the phases, Sufficiency and Relevance were rated the highest, with means of 3.88 and 3.84, respectively, reflecting the experts’ perception that the activities are appropriate and relevant. However, the Clarity criterion showed greater variability (SD = 0.261), suggesting that the presentation of activities could benefit from further precision to facilitate understanding.

The boxplots (Figure 5 and Figure 6) illustrate these findings, showing how the phase scores cluster around the maximum value of 4.0, reflecting a generally positive evaluation. However, the greater dispersion in the Evaluation phase and in the Clarity criterion highlights the need for adjustments to improve consistency and understanding of the activities, ensuring that evaluators interpret the procedures uniformly across all phases of the self-assessment process.

In addition to the analyses described above, as a result of an overall evaluation of the guide (

Table 8. Results of the assessment of the guide according to the criteria.

						Shapiro–Wilk
Dimensions	Media	Medium	DE	Min	Max	W	p
Criteria
Sufficiency	3.81	3.89	0.193	3.33	4.00	0.873	0.013
Coherence	3.77	3.80	0.239	3.05	4.00	0.850	0.005
Relevance	3.80	3.87	0.222	3.13	4.00	0.829	0.002
Clarity	3.67	3.67	0.228	3.10	3.97	0.944	0.289

Note: Low W values and p-levels of less than 0.05 (<0.001) from the Shapiro–Wilk test indicate that the data do not follow a normal distribution, suggesting that non-parametric statistical methods should be considered for further complementary analysis.

), the criteria of Sufficiency, Relevance, and Coherence were consistently well-rated by the experts, with averages above 3.75, reflecting a positive perception. This suggests that the experts consider the proposed guide to be adequate, relevant, and coherent, with moderate variability in responses, indicating consistency in the interpretation and application of these criteria.

On the other hand, the Clarity criterion showed greater dispersion (SD = 0.228) and a slightly lower average (3.67), with several outliers, suggesting differences in interpretation (Appendix A Figure A1), which highlights the need for adjustments in the presentation of certain aspects of the guide to ensure greater understanding and uniformity among evaluators. This result confirmed previous findings, which identified the need for adjustments to improve comprehension.

Additionally, although Figure 5 and Figure 6, Appendix A, and Figure A1 show outliers, these are within high and acceptable values, suggesting that while there is some variability in the responses of certain evaluators, these values do not represent a negative evaluation. The presence of these outliers may be due to subtle differences in the perception of certain aspects, but they remain within the boundaries of a favourable evaluation.

From these findings, the guide was modified, resulting in a new version that would be used in the pilot study. Five new requirements were added to the model, increasing the total from 48 to 53, with corresponding pieces of evidence that would guide the aspects to be assessed. Additionally, for each dimension, its objective was added, and for each standard, its objective and the respective qualitative rating scale were included, guiding the evaluator in determining the score. A description was also added explaining the aim of each requirement, and each requirement was complemented with an association with the relevant pieces of evidence. An example of an expanded requirement is provided in Appendix A,

Table A4. Statements of a requirement of the final self-assessment template.

	Description or Statement
Requirement	4.2.2. The e-learning platform ensures universal availability and accessibility of the programme or course for both teachers and students.
Description of the requirement a	This requirement aims to make the learning platform universally accessible and available to all users, ensuring that both students and teachers can access course content and related services at any time and from anywhere.
Evidence of the requirement a	Documents evidencing the availability and accessibility of the platform, records of accessibility audits, incident reports, and their solutions.
Examples of actions a	Implementation of regular accessibility testing to ensure that the platform complies with international standards, such as WCAG. Maintaining an infrastructure that guarantees close to 100% uptime, minimising interruptions and ensuring continuous access to course resources.
Standard to which it belongs	4.2 Learning management platform: the institution has an accessible IT platform to support the virtual training process and administrative management.
Objective of the standard a	Ensure that the institution provides a learning management platform that is accessible, reliable, and effective, facilitating the e-learning process and academic and administrative management in a way that meets the needs of all users, including those with disabilities.
Rating scale of the standard a	Satisfactory compliance: The learning management platform is fully accessible and continuously available to the entire university community, supporting both teaching-learning and academic and administrative management. All requirements are met without limitations, allowing for a seamless and barrier-free user experience. Partial compliance: The platform is functional and meets most requirements, but there may be areas where accessibility or availability requires improvement. Some functions may not be fully accessible to all users or availability may have been occasionally interrupted. Poor compliance: The platform has significant shortcomings in terms of accessibility or availability, which affects users’ ability to access courses and administrative formalities. The limitations can lead to difficulties in the teaching–learning process. Non-compliance: The platform is not suitable or accessible for the e-learning process. The lack of availability or significant accessibility creates significant barriers for users, compromising the effectiveness of e-learning.

^a Attributes added to the model to improve clarity and target applicability.

.

4.3. Pilot Study

As a result of objective one of the pilot study phase (O1), one of the findings of the self-assessment pilot was significant variation in the weighting of standards and requirements, reflecting the model’s adaptability to different institutional realities.

Table 9. Comparison of the weightings assigned to the standards for each institution.

Standards	Institution 1	Institution 2	Institution 3	Institution 4
1. Organisation
1.1 Organisation	Critical	Critical	Critical	Important
1.2. Course or academic programme information	Critical	Important	Critical	Important
1.3 Economics and technology financing	Critical	Critical	Critical	Critical
1.4 Knowledge management	Important	Important	Critical	Critical
1.5. Research and innovation	Important	Important	Important	Critical
2. Students
2.1 Student support	Critical	Critical	Critical	Important
2.2 Admission	Critical	Critical	Critical	Critical
2.3 Diversity and inclusion	Critical	Critical	Critical	Critical
3. Teaching
3.1 Teacher profile	Critical	Critical	Critical	Critical
3.2. Support for teachers	Important	Important	Critical	Critical
3.3 Learning content and resources	Critical	Critical	Critical	Critical
3.4 Learning strategies	Critical	Critical	Critical	Critical
3.5 Electronic evaluation	Critical	Critical	Critical	Critical
4. Infrastructure
4.1 Technological infrastructure and equipment	Critical	Critical	Critical	Critical
4.2 Learning management platform	Critical	Critical	Critical	Critical
4.3 Technical assistance and support	Critical	Important	Critical	Critical

and

Table 10. Distribution of weights assigned to the requirements for each institution.

Level	Requirements
	Institution 1	Institution 2	Institution 3	Institution 4
Critical	39	33	45	8
Important	11	14	6	44
Desirable	2	3	1	1
Not applicable	1	3	1	0

list the results obtained. Table 9 highlights significant differences in how institutions prioritised the standards across the four dimensions. This prioritisation was conducted using a Likert scale, which allowed for the evaluation of the importance assigned to each item, with three categories: “Critical”, corresponding to an essential requirement that, if not met, would severely compromise the quality of the standard; “Important”, which has a significant but not crucial influence; and “Desirable”, which improves the process quality, although its absence does not severely compromise compliance with the standard.

For example, Institution 1 and Institution 3 assigned a critical weighting to most of the standards across all the dimensions, reflecting a focus on the urgency of meeting almost all the evaluated aspects. In contrast, Institution 4 tended to rate more standards as important, especially in the Student Body and Teaching dimensions, indicating a differentiated perspective on which areas require priority attention.

Table 10 shows a summary of the weighting distribution assigned at the levels of the evaluated requirements. Institution 1 and Institution 3 weighted most of the requirements as critical (39 and 45, respectively), indicating a strong focus on assessing key standards. In contrast, they designated fewer requirements as important or desirable, suggesting that they perceive less flexibility in the critical aspects of the evaluated process. Institution 4, on the other hand, only weighted eight requirements as critical, and the vast majority (44) as important, highlighting greater flexibility in their evaluation, without compromising quality but considering certain areas as more susceptible to adjustments or gradual improvements. Finally, Institution 2 adopted an intermediate position, with 33 requirements weighted as critical and 14 as important, reflecting a balance between what they consider essential and what can be improved without critically affecting quality. Additionally, two institutions identified requirements as not applicable to their institutional reality.

This analysis confirms the premise of the guide that the evaluation of each standard and requirement should be adjusted to the institutional reality and the specific context of each organisation. The flexibility in the weighting allows each institution to adapt the evaluation model to its circumstances, facilitating a more relevant self-assessment process.

Another finding, aligned with the second objective of this phase (O2), revealed significant variations in the level of compliance with the proposed quality standards across the dimensions of Organisation, Student Body, Teaching, and Infrastructure (see

Table 11. Comparison of the weightings assigned to the standards for each institution.

Standards	Institution 1	Institution 2	Institution 3	Institution 4
1. Organisation	0.74	0.52	0.89	0.9
1.1 Organisation	0.72	0.57	0.94	0.94
1.2 Course or academic programme information	0.68	0.76	0.75	0.83
1.3 Economics and technology financing	1	0.35	1	1
1.4 Knowledge management	0.47	0.51	0.9	0.82
1.5 Research and innovation	0.74	0.44	0.87	0.9
2. Students	0.51	0.52	0.87	0.77
2.1 Student support	0.49	0.72	1	0.86
2.2 Admission	0.35	0.54	0.9	0.77
2.3 Diversity and inclusion	0.7	0.3	0.7	0.7
3. Teaching	0.23	0.4	0.55	0.83
3.1 Teacher profile	0.15	0.41	0.6	0.85
3.2 Support for teachers	0.43	0.58	0.9	0.8
3.3 Learning content and resources	0.11	0.24	0.12	0.68
3.4 Learning strategies	0.3	0.7	0.65	0.85
3.5 Electronic evaluation	0.23	0.12	0.49	0.93
4. Infrastructure	0.74	0.61	0.53	0.8
4.1 Technological infrastructure and equipment	1	0.57	1	1
4.2 Learning management platform	0.72	0.85	0.3	0.7
4.3 Technical assistance and support	0.5	0.3	0.3	0.7

Note: The possible rating scale is between 0 and 1, where 1 represents the highest compliance. These values correspond to the qualitative ratings: “satisfactory compliance” (1.0), “partial compliance” (0.7), “poor compliance” (0.3), and “non-compliance” (0.0).

). Overall, both areas of strength and critical points that require attention were identified to ensure better quality in terms of accessibility and inclusion in virtual education analysis, confirming the guide’s premise that the assessment of each standard and requirement should be adjusted to the institutional reality and the specific context of each organisation. Flexibility in weighting allows each institution to adjust the assessment model to its circumstances, facilitating a more relevant self-assessment process.

Regarding the Organisation dimension, Institutions 3 and 4 reflect a high level of compliance with ratings of 0.89 and 0.9, respectively. Both institutions highlighted their financial support in addressing the needs for technological upgrades and processes related to virtual education. Conversely, Institutions 1 and 2 showed moderate or partial compliance, with ratings of 0.74 and 0.52, suggesting a need to strengthen institutional strategies to support virtual education. At Institution 2, the standard related to financial investment was rated poorly. Although the need to strengthen virtual education was identified at the planning level, being a public university, its investment capacity depends on government allocations, which has limited its ability to respond.

On the other hand, the Student Body dimension revealed a high level of compliance at institutions 3 (0.87) and 4 (0.77), with a clear focus on student support and accessible admission processes. In contrast, Institutions 1 (0.51) and 2 (0.52) showed partial compliance in this dimension, mainly due to a lack of a clear focus on accessibility and inclusion.

In the Teaching dimension, Institution 4 stands out again, with a satisfactory level of compliance (0.83), demonstrating a strong focus on faculty profile and the use of effective and accessible learning strategies. Institution 3, although it had partial compliance (0.55), showed deficiencies and areas for improvement, mainly related to electronic assessment and the quality of educational content. Meanwhile, Institutions 1 (0.23) and 2 (0.4) exhibited critical deficiencies in this dimension, reflecting an urgent need for improvements.

Finally, in dimension 4, Infrastructure, both Institution 4 (0.8) and Institution 1 (0.74) achieved satisfactory compliance, indicating that they have robust platforms and adequate support for virtual teaching. However, Institutions 2 (0.61) and 3 (0.53) showed deficiencies, especially in technical support, with a low capacity to meet the technical needs of both teachers and students observed, negatively affecting the educational experience.

In summary, Institutions 3 and 4 demonstrated good alignment with the self-assessment model standards, while Institutions 1 and 2 had significant areas for improvement, particularly in the Teaching and Student Body dimensions. The self-assessment results provide a clear basis for planning corrective actions to strengthen the quality of virtual education.

As an individual example, the results of Institution 2 are presented (Table 11). Significant differences in the level of compliance with the model can be observed. The Organisation dimension shows an overall compliance of 52%, with standards such as 1.3, 1.4, and 1.5 evaluated as deficient, indicating significant areas for improvement, particularly in financial matters. The highest-rated standard was 1.2. Information on the course or academic programme was rated as partial compliance, at 72%, which, while indicating good management of information, is deficient in terms of the accessibility of this content.

The Student Body dimension achieved an overall rating of 52%, standing out in “student support and guidance” (72%, partial compliance), reflecting a commitment to student support, but with weaknesses in “diversity and inclusion”, evaluated with a deficient compliance of 30%, due to the lack of formally established and applicable processes at the institutional level that promote educational inclusion.

The Teaching dimension presented one of the most significant challenges, with an overall compliance of 40%. While “learning strategies” were satisfactorily applied (70%), the standards “learning content and resources” and “electronic assessment” were rated as non-compliance, reflecting the urgent need to work on these aspects.

Finally, in the Infrastructure dimension, with a compliance of 61%, there was good management of the “learning management platform” (85%, satisfactory) but poor “technical assistance and support service”, suggesting that, although there is a solid infrastructure, the support services require restructuring to improve the online educational experience.

A better interpretation and quicker identification of strengths and weaknesses (improvement priorities) can be seen in Figure 7, through a set of radar charts.

5. Discussion

The process followed in this study stands out for its robustness and methodological rigour, classifying it as a “methodological research design” study [63]. This methodology is characterised by a systematic, organised, and iterative approach to collecting information and analysing the gathered data, setting it apart from other approaches in several key aspects. Firstly, the procedure employed is supported by a dual validation process, with feedback between phases, resulting in improved versions of the guide, ultimately leading to the application of the studied product, thus ensuring a more comprehensive and exhaustive approach.

Compared to other studies on the validation of educational tools, the methodology employed in this study stands out for its holistic approach, combining both quantitative and qualitative techniques. The use of internal consistency measures such as Cronbach’s Alpha and McDonald’s Omega allowed us to assess the reliability of the instrument, confirming that the items were consistent within the areas assessed.

The analysis of Aiken’s V and the ICC provided a more detailed perspective on content validity and the agreement among the experts, validating the idea that the items adequately reflected the quality aspects of a self-assessment in virtual learning institutions. Additionally, the use of central tendency measures (mean, median, and standard deviation) along with box plots helped to observe general response patterns and identify the spread and outliers in the ratings, offering a clearer view of the variability in the evaluators’ perceptions.

These quantitative analyses were complemented by a thematic analysis of the qualitative observations, which provided a more detailed focus on the applicability and clarity of the items assessed. These observations were key to identifying aspects of the guide that, although statistically validated, required improvements to facilitate their understanding and applicability in real-world contexts.

The iterative feedback process between the phases ensured that the results obtained were integrated into improved versions of the guide, so that each new version reflected the necessary adjustments to respond to the particularities of the experts’ perceptions.

A key aspect of this study was the differentiation of the participating experts in each validation phase, allowing for a broader range of perspectives and strengthening the robustness of the results. In the first phase, four experts participated, while in the second phase, the group was expanded to 20, which not only increased representativeness but also helped to mitigate potential biases, such as “recall bias”. By changing the evaluators between the phases, independent responses were ensured, resulting in a more objective validation process.

The results of this study have significant theoretical and practical implications. From a theoretical perspective, this study contributes to the literature on quality assessment in education by providing an empirically validated self-assessment model. Moreover, it demonstrates a comprehensive methodological approach that combines quantitative and qualitative validation techniques, offering a robust framework that can serve as a reference for future validations of tools related to education or other fields. Additionally, by incorporating accessibility as a central criterion, this work expands traditional models of quality in education, aligning them with the principles of inclusion and equity in SDG 4.

In terms of practical implications, the proposed self-assessment model provides educational institutions with a versatile and reliable tool for evaluating quality in virtual education through an accessibility-focused lens. This enables quality managers to identify areas needing improvement and devise strategies tailored to the specific needs of various stakeholders and their unique educational contexts.

These implications position this model as a valuable contribution to the literature on educational quality while also making it an applicable tool that can be used to strengthen accessibility and quality standards in virtual education. In doing so, the model promotes inclusive and sustainable practices in higher education institutions.

6. Conclusions

The main findings of each phase and their relevance for the future implementation of the model are discussed below. Additionally, the limitations of the study and future work are discussed.

6.1. Preliminary Validation

From the preliminary validation, the analysis of Aiken’s V index indicates that the guide is perceived very favourably in terms of its practical value, content, structure, and configuration. Specific recommendations were received through the evaluation rubric, which allowed the guide to be refined, ensuring that it meets high standards of quality and relevance in the field of virtual education.

While the evaluators agreed on the quality and usefulness of the guide, there was less of a consensus regarding the weighting and rating of the standards and phases. To address this issue, it was decided to give the self-assessment team greater flexibility, allowing them to adjust the ratings according to the institutional context. Consequently, the need for a second content validation, an extended validation, was reinforced to ensure greater consistency and acceptance of the guide before its application or implementation in real-world contexts.

6.2. Extended Validation

The evaluation of the internal consistency of the instrument through the Cronbach’s Alpha and McDonald’s Omega coefficients revealed a high degree of reliability, with coefficients above 0.80. This reinforced the robustness of the questionnaire, ensuring that the items within each dimension and each phase consistently assessed the proposed constructs.

Furthermore, high content validity was shown in most of the evaluated items, with Aiken’s V scores above 0.80, indicating a high degree of consensus among the experts. However, some of the items related to clarity showed greater variability, suggesting that certain content could benefit from adjustments to improve the guide’s understanding and applicability.

Additionally, the experts made valuable qualitative contributions to improve the guide. Some of the observations included suggestions to improve the clarity of the statements and better specify the requirements and evidence.

In terms of comparing the evaluations, the use of boxplots helped to identify patterns of consistency and variability. The results showed how the medians in most of the evaluated items approached the maximum values of four points, suggesting that the experts generally gave a positive evaluation. However, significant variability was noted in some of the evaluated criteria, particularly concerning clarity. This suggested that certain points in the guide could benefit from improvements, particularly in the formulation and clarity of certain items.

In summary, the extended validation confirmed the robustness of the self-assessment model, highlighting its coherence, relevance, and sufficiency, but also pointed to areas where clarity could be improved to ensure more effective and uniform application in diverse contexts. These adjustments will ensure that the model is not only relevant, but also easily applicable and understandable for implementing institutions.

6.3. Pilot Study

The flexibility of the model, as evidenced by the differences in the weighting assigned to the various standards and requirements in each institution, confirms and validates the need to adjust the scoring scale according to the specific needs of each institution.

Furthermore, regarding the application results, the analysis points to the high applicability of the self-assessment model in real contexts. The developed tool proved to be a key element in obtaining a diagnosis of its performance in relation to compliance with the proposed quality standards, creating a space for critical analysis that contrasts the obtained scores with the institutional reality and considers improvements based on the identified weaknesses.

In essence, the pilot application confirms that the guide is not only applicable in various institutional contexts but can also contribute to improving internal quality processes, as long as it remains open to feedback and adjustments according to the specific needs of each institution.

6.4. Limitations and Future Work

Overall, the validation process and the pilot application confirm that the self-assessment model is not only robust but also adaptable and applicable to different institutional contexts. While the findings are positive, this study has some limitations that need to be considered:

(a)

Sample size: Although the number of experts involved in the validation was relatively small, this limitation was influenced by the difficulty of recruiting experts with relevant experience. The inclusion criteria required the participants to have experience in quality assessment processes at the institutional level, which, while restricting the availability of experts, ensured the quality of the assessments. However, this limits the ability to generalise the results to a broader spectrum of institutional realities. Future studies could expand the sample to include experts with different levels of experience, allowing for a more diverse and generalisable perspective.

(b)

Scope of the pilot study: The implementation of the model was relatively short, covering only the phase of identifying strengths and weaknesses that require attention. This prevented the observation of its long-term impact on the institutions’ internal quality processes. Future work could conduct longitudinal studies to analyse how institutions are integrated into continuous improvement.

(c)

Survey fatigue: Despite the measures implemented to mitigate respondent fatigue due to the length of the extended survey questionnaire (165 possible responses, 126 quantitative and 39 qualitative), the impact of survey fatigue cannot be completely ruled out. This limitation may have influenced the accuracy or consistency of some responses.

In the future, a broader application of the model is recommended to obtain more generalisable data and to continue refining items in which clarity or specificity could be improved. This extension (n > 100) would allow for a confirmatory factor analysis (CFA), which would empirically validate the structure of the proposed model, ensuring that the measured dimensions and standards adequately fit the theoretical constructs, optimising its application on a large scale.

The results obtained from the validation process ensured the reliability and validity of the guide, highlighting the importance of accessibility as a central element in the evaluation of quality in virtual education. In this way, the guide not only seeks to guarantee the quality of education or training but also provides institutions with an adaptable tool that responds to accessibility needs, which is key in the context of virtual higher education.

Despite the limitations, this study revealed a solid and replicable methodology for other contexts. Similarly, it is crucial for psychometric strategies to be considered in the design of assessment tools to ensure reliability and validity. Additionally, constant feedback and adjustments based on each institution’s specific needs will continue to be key elements in ensuring the model’s relevance and effectiveness in improving educational processes in virtual higher education.

Addressing the main research question, this study demonstrates the feasibility of consolidating and validating a self-assessment guide for quality in virtual education from an accessibility perspective, adaptable across diverse institutional contexts. The model’s inherent flexibility allows it to be tailored to the circumstances of each institution, thus providing an adaptable, accessibility-focused tool that is especially pertinent to virtual education.

Finally, with respect to the research sub-questions, the findings affirm that the model’s structural elements are both effective and suitable for facilitating its implementation in accessible and equitable virtual education environments. This was evident through high levels of expert consensus and consistency, as well as the modifications made to enhance clarity, which further improved its applicability in various institutions. Furthermore, the model exhibited a solid scientific foundation and robust content, supported by analyses of internal consistency and criterion validity conducted during the preliminary and extended validation phases. Together with insights from the pilot study, these results support the model’s practical adaptation across diverse institutional settings.