Quality Assurance Framework for the Design and Delivery of Virtual, Real-Time Courses

Abstract

Designing and delivering outcome-based courses that emphasize learner-centric educational discourse and active learning is challenging, especially in online learning environments. Ensuring quality in the design and delivery of such courses in the virtual space requires a well-defined framework with key constituents that interact based on ordered sequences of events. Despite the pressing need for a quality assurance system for today’s virtual, real-time courses, such a system has not been systematically designed. A coherent quality assurance system requires a clear framework that defines the interacting constituents. This work proposes a conceptual and generic “Quality Assurance” (QA) framework, based on experiences primarily in Science, Technology, Engineering, and Mathematics (STEM) fields, for the effective design and delivery of outcome-based virtual, real-time courses that incorporate active learning practices. This Quality Assurance framework may be adjusted to serve as a blueprint that, once adjusted by institutions to accommodate their missions, guides institutions in developing or amending their policies and procedures for the design and delivery of virtual, real-time courses; in addition, such a framework is important for institutions to develop Quality Assurance systems that integrate mechanisms for continuous improvement. The proposed quality assurance framework includes three constituents: a “Teaching and Learning Support” (TLS) that trains educators on pedagogical approaches and the capabilities of the institution’s Learning Management System (LMS); an “Information and Communication Technology Support” (ICTS) that assists educators with the technologies and tools available in the learning management system; and a “Course Management System” (CMS) that encapsulates course design, delivery, and assessment; this study focuses primarily on this “Course Management System” constituent.

1. Introduction

The coronavirus (COVID-19) pandemic has forced academic institutions to rapidly shift from a face-to-face course delivery mode that is usually supported by a Learning Management System (LMS) to virtual, real-time online classes. Institutions sometimes integrated their LMS with other platforms, like Microsoft Teams or Zoom, to foster interactions in the virtual, real-time lessons. These interactions included web conferencing, video streaming, instantaneous sharing of documents, one-on-one or one-to-many dialogues, and real-time participation in group activities. Unfortunately, most educators who were forced to engage in this sudden technological shift were unprepared to effectively engage their students, especially those using pedagogical principles that include active learning practices and outcome-based lesson design, delivery, and assessment [1,2].

The teaching and learning landscape expected to experience a steady increase in academic programs utilizing virtual, real-time classes: online teaching and learning methods, pedagogies, and tools will become more prevalent. Indeed, blended and virtual real-time learning are progressively gaining popularity and becoming an integral part of academic curricula [3,4]. The virtual, real-time session portion of blended learning activities that use gamification as an active learning practice may be able to create an “edutainment” environment that promotes and sustains the motivation to learn [5,6,7,8]. Blended learning may even be able to provide opportunities for collaborative learning, for example, through pairing-and-sharing and group learning [9].

The poor-quality and low-standards stigmas occasionally associated with e-learning, and the perceived technological complexity of operations, require the establishment of a Quality Assurance (QA) system for virtual, real-time classes [10]. A QA system, especially in virtual, real-time education, provides a foundation for systematic diagnosis, decision-making, and strategic planning to continuously improve online instruction [11]. A QA system also ensures that the academic goals of virtual, real-time instruction are met, implemented, and evaluated in an accountable and transparent manner [12]. In the context of modern educational paradigms, any QA system should include pedagogies that implement active learning practices whose efficacy is assessed by learning outcomes.

Recent work provides recommendations for developing online courses; this work highlights emerging trends, best-practices, and challenges in implementing engaging and effective online courses that maintain student success, and motivate learners [13]. The recommendations address several inter-related issues, including teacher education on emerging technologies, current teaching and learning practices, effective online instructional strategies, and student accessibility, equity, and success.

1.1. Educational Pedagogies

When designing class material, educators must strategically address both the product (lesson plans or instructional material) and the process (delivery methods, activities, implementation and management of the overall design process) [14,15]. Similar to any other process, designing class material and learning methods, conducting real-time delivery activities, and assessing instructional success occur in a sequence of phases involving decision-making and problem-solving [16,17]. In outcome-based education, designing class material and learning methods is associated with “clear learning objectives, carefully structured content, relevant student activities, and assessment strongly tied to desired learning outcomes” [18]. When blending active learning practices with outcome-based learning, the “relevant student activities” translate into appropriate active learning techniques [18].

Designing and delivering class content plays an essential role in virtual, real-time courses [19,20]; these include rich and flexible educational experiences involving the integration of combinations of a diverse array of tools [21,22,23]. As such, virtual, real-time teaching adds a layer of technological difficulty to traditional in-class delivery modes; this difficulty is exacerbated by the inclusion of outcome-based learning that incorporates active learning practices.

1.1.1. Outcome-Based Education

Outcome-based education is a student-centric educational theory that seeks to align outcomes with course pre-requisites, content, instruction, learning activities, and assessment, and is practiced around the world after gaining international recognition pursuant to the signing of the Washington Accord in 1989 [24]. Indeed, reputable accreditation agencies, such as the Accreditation Board for Engineering and Technology (ABET), emphasize what needs to be taught, how it should be taught, and what is being learned [25]. Outcome-based education focuses on shaping an educational system around what is essential for all students to be capable of successfully performing at the end of their learning experiences. Successful outcome-based education programs demonstrate that students have attained the knowledge, attributes, and skills required to be successful after graduating from their educational institutions.

There are multiple instructional strategies in outcome-based education. These strategies are based on the realization that to attain the outcomes, educators need “to set up an environment that maximizes the likelihood that students will engage in the activities designed to achieve the intended outcomes” [26]. Four principles of outcome-based education have been proposed [27]. First, educators must center planning and teaching on the pre-determined outcomes. Second, educators should design useful learning material and activities that can be traced back to desired pre-determined outcomes [28]. Third, educators should set challenging tasks that are appropriate for the students’ levels to encourage continuous and deeper learning [29]. Fourth, educators must realize that a variety of learning methods must be deployed to target the majority of students [30].

Instructional strategies for outcome-based virtual learning are still being researched. Indeed, qualitative research on outcome-based, blended learning reported inadequate preparation from faculty members because of their lack of confidence in their grasp of the technology and their inexperience in the online environment [3]. The scarcity of studies on instructional strategies for outcome-based, virtual, real-time classes may be attributed to the relatively recent, if not abrupt, reliance on this platform as the predominant tool of instruction.

1.1.2. Active Learning

Active learning is “a method of learning in which students are actively or experientially involved in the learning process and where there are different levels of active learning, depending on student involvement” [31]. When students engage in active learning, such as working together to apply a newly learned skill, they are more likely to retain what they have acquired [32]. Active learning strategies are an integral part of outcome-based education and have positively impacted learning outcomes [33,34]. For instance, class debate as an active learning method was shown to significantly enhance student attainment of course learning outcomes [35]. Indeed, some engineering higher education programs have worked on clearly defining how learning outcomes relate to active learning methods [36].

Instructional strategies that involve students as active participants in the classroom are challenging; these challenges are amplified in a virtual space where online tools are the primary means of engaging students. An evaluation of low- and high-tech active learning classrooms concluded that several problems emerged with high-tech active learning, most of which were related to the instructors’ experiences and the extent of their abilities to deal with technological obstacles [2]. Another study examined the use of student-generated audio files as an active learning technique within the context of online, asynchronous course delivery [37]; the goal was to gain an understanding of how this approach impacted the students’ perceived satisfaction, engagement, connectedness, learning, and utilization. Indeed, such active learning approaches increased the students’ learning and assisted students in effectively connecting and communicating with peers.

Some institutions provide guidelines on the use of active learning methods in virtual, real-time courses. Some institutions have even advocated the incorporation of active learning strategies in Science, Technology, Engineering, and Mathematics (STEM) introductory courses [38]. Periodic training may be needed for educators to acquire and/or update skillsets in implementing diverse and evolving active learning strategies; this includes training in classroom planning, choosing learning activities, using technologies, and utilizing appropriate assessment tools [39].

The proposed QA framework for the design and delivery of virtual, real-time courses integrates outcome-based learning and active learning practices.

1.2. Quality Assurance

In the education sector, especially in higher education, stakeholders, educational institutions, and students hold different conceptions of “quality”: as posited, “quality is in the eye of the beholder” [40,41]. This presents challenges for the development of standardized assessment methodologies that meet the criteria for public accountability, academic credibility, practical feasibility, and audit of “quality” [40]. Higher education institutions could use external QA processes to sustain their development [42]; however, these processes have to be carefully designed to conform to the missions of the institutions. Leadership that is committed to ensuring a safe and creative college environment is essential for designing effective QA processes [43]. Indeed, studies in higher education in Nigeria have shown that for the attainment of quality higher education, any QA system must include physical facilities and infrastructure, innovations, quality teaching, human resources, and curriculum innovation that conform to market demand [44].

Many QA studies have highlighted the importance of establishing various educational and operational systems. QA studies in higher education programs revealed the importance of developing skills in syllabus design, for example through focused workshops, university guidelines, and teacher’s assessments [45]. A QA study that was focused on online courses recommended the importance of establishing the “Office of Instructional Design” to ensure quality design and delivery of online courses, and to train educators prior to engaging in distance teaching [46]. Another study encouraged the development of a quality culture to improve curriculum design, enhance internal quality work, and support documentation for external QA auditing [47]; such a quality culture would also engage educators and other internal stakeholders in the development of a range of educational programs, thus promoting a system for improved quality in higher education institutions.

QA studies have also examined how institutional operations influence planning. For example, one study recommended the development of QA policies focused on accreditation to cement the relationship between university ranking and the concept of quality [48]. An extensive literature review was conducted to examine key challenges of QA in European higher education institutions [49]; these key challenges included educational systems, quality systems, quality cultures, funding, and accreditation agencies. This study showed that many institutions still struggle today with the challenges associated with converging all of the key challenges in one QA system.

In higher education, QA is viewed primarily as continuous improvement that is driven by mechanisms and assessments. There seems to be no agreement on a universal definition, model, and/or practices of QA in higher education; indeed, having a universal definition may be unnecessary since quality and QA tend to be contextual, that is, defined by the cultures of the institutions [50,51]. As a result, institutions develop their own definitions, models, and evaluation processes that suit their respective cultures and stakeholders.

QA processes can be established at the macro or micro levels. Macro-level QA processes manage institutional oversight; a micro level QA process may manage student experiences, such as offering support in orientating students to an online learning environment through the types of engagement tools being offered in the course delivery method [52]. Achieving quality teaching and learning is an important and complex effort involving multiple interacting levels and dimensions; these include managing the technological educational infrastructure, designing curricula and courses, training educators, and preparing students [53].

The rapid growth of online and blended academic programs in higher education has led to the growth and diversification of instructional methods; institutions responded by developing their own processes and implementing strategies to ensure that these new forms of delivery supported the value, quality, and validity of their higher education certifications. While most institutions have implemented some form of internal, self-regulated QA procedures for e-learning, a comprehensive and practical QA framework that systematically covers inputs, processes, and outputs in online learning has not been formally posited [54]. Indeed, a successful QA system needs to be sufficiently flexible to adapt to differentiated approaches and innovations in pedagogy [55]; in addition, the effective implementation of such a system requires significant effort, resource commitment, and internalization to develop quality awareness and values for continuous improvement.

A QA toolkit has been developed that is based on a broad suite of frameworks, rubrics, assessment criteria, and systems in online and blended learning in higher education [56]; this toolkit can help guide the development of consistent approaches to the QA of online education at the policy, regulatory, and institutional level. This holistic, domain-based toolkit promoted a culture of quality within the global education sector. It established a range of criteria to help QA agencies assess online and blended programs, and assisted academic institutions in refining their key performance indicators and improving their standards for online education [56].

A comparative study suggested that although a uniform approach is absent, there are quality standards that institutions can effectively implement to address QA in online learning [52]. A QA outline that is template- and checklist-driven was proposed that is dynamically and iteratively intertwined with the e-learning development process [57]. This proposed outline defines the e-learning development phases and recommends practical steps for each phase.

A professional development cycle that intertwines virtual, real-time class instructions with related QAs and mechanisms for monitoring the cycle to guide continuous improvement has been proposed [58]; key aspects of this professional development system include how to plan online activities, construct objectives, and construct assessments that are used to improve instruction. Indeed, virtual, real-time classes require significant technical development such that most educators planning to engage in this mode of delivery must benefit from comprehensive training on their institution’s LMS and other related available tools [59]. These skills are necessary for class management, advising students on the use of the LMS, and manipulating the LMS features during class delivery. A comprehensive QA framework should incorporate these professional and technical developments within a system that includes support mechanisms for educators and students.

This work incorporates different levels of QA and proposes an institution-wide QA framework for the design and delivery of virtual, real-time courses using current pedagogical principles. The proposed generic QA framework provides a roadmap for educators and institutions to implement instructional programs that are outcome-based and rooted in active learning practices.

2. Method Used to Design the Quality Assurance Framework

The ultimate goal of these studies is to design a QA system for virtual, real-time courses that incorporate outcome-based education and active learning practices. A literature review was carried out that queried the following databases: Academic Search Ultimate; Education Research Complete; Professional Development Collection; Taylor and Francis Journals; Emerald Journals; Sage Journals; and ScienceDirect. Keywords, such as “quality assurance”, “outcome-based education”, “active learning”, “interactive learning”, “virtual teaching”, “online learning”, “blended learning”, “edutainment”, and similar terms were used: the results of this review are outlined in the Introduction. The QA framework proposed in this work is conceptual and was developed based on this review and the experiences of the authors in higher education, primarily in STEM disciplines. The main goal of the conceptual QA framework is to integrate the following three elements:

• Outcome-based education
• Active learning practices
• Virtual, real-time teaching and learning

The proposed QA framework is crucial for the development of systems that ensure quality in the design and delivery cycle of courses. In addition, this study focuses on one constituent of this proposed QA framework expanding it into a sequence of modules and their phases; these could aid institutions and their educators in designing and delivering outcome-based courses that incorporate active learning practices in a virtual, real-time learning environment.

3. Conceptual QA Framework

A QA framework for the design and delivery of courses is based on three interacting constituents. The term constituent is used to represent a conglomeration of activities that include support, training, design, delivery, and/or assessment. As such, constituents do not have to correspond to discrete units/offices/departments in universities, although that correspondence is an option. The three constituents of the QA framework are (Figure 1):

• An institution “Information and Communication Technology Support” (ICTS) constituent that manages the Learning Management System (LMS) and provides support to educators during the design and delivery of courses.
• An institution “Teaching and Learning Support” (TLS) constituent that trains educators in outcome-based teaching and active learning methods. Training of educators and students in the institution’s LMS is ideally managed by both the ICTS and TLS where the TLS focuses on active learning capabilities of the LMS. In addition, TLS train educators in developing effective communication skills in an online environment.
• A “Course Management System” (CMS) constituent that guides the sequential actions of educators for the design, delivery, and direct assessment of outcome-based, virtual, real-time classes that incorporate active learning practices; this constituent also provides a system for the indirect assessment of the performance of educators.

The effective design and delivery of courses requires cooperation that is based on clear processes between the three constituents. The QA framework would guide the development of policies and procedures for each of the three constituents with their related quality assurance standards and requirements. The establishment of a QA framework ensures the efficacy of the current teaching cycle, and ideally, establishes procedures for the continuous improvement of programs and teaching practices.

This work focuses primarily on the “Course Management System” (CMS) constituent of the QA framework, and provides sequential guidelines for the design, delivery, and assessment of virtual, real-time courses that incorporate outcome-based and active learning practices. The involvement of the other two constituents, ICTS and TLS, are highlighted, where appropriate.

4. The Course Management System

The Course Management System (CMS) constituent of the QA framework includes three modules that are depicted in Figure 1. Each of these modules has sequential phases that are depicted in Figure 2. It should be noted that educators are indicated as the major stakeholders of the CMS constituent because this article focuses on the design and delivery of individual courses. However, courses typically exist in the context of programs/curricula that include co-dependencies, for example, co-requisites and pre-requisites. These should be taken into account when assessing the performance of students and educators, as highlighted below. In this context, assessment and adjustments to courses should involve the academic units of the educators, for example departments, who manage the relationships between courses in programs/curricula and the interactions between the three constituents, CMS, ICTS, and TLS.

4.1. Module 1—Outcome-Based Lesson Preparation and Rehearsal

The first module of the CMS encapsulates all the training, as indicated in the phases below, of educators in the educational and technological tools required for the effective design and delivery of virtual, real-time courses; it also includes all the preparatory tasks that educators must complete prior to a class session. This outcome-based lesson preparation and rehearsal module includes a sequence of four phases (Figure 2).

4.1.1. Module1/Phase1: Design Class Material, Learning Methods, and Direct Assessments

Design Structure

Designing outcome-based class material, learning methods, and direct assessments involves primarily the Educator (E), ICTS (I), and TLS (T). During this phase, educators design the class material using the following general protocol (Figure 3):

Educator Preparation

• Learn about the capabilities of the institution’s LMS; typically, this is provided by the institution’s ICTS. This training provides educators with the tools, and limitations, for the virtual, real-time delivery mode of teaching.
• Learn about different active learning methods; typically, this is provided by the institution’s TLS. For the virtual, real-time mode of delivery, the focus should be on the active learning methods that can be implemented through the institution’s LMS; educators can be trained on these LMS features by TLS in collaboration with ICTS. Two active learning tools, and some ways they could translate from face-to-face to online modes of delivery, are shown as examples in

  Table 1. Sample of active Learning Management System (LMS) tools ^a.

  In-Class Activity	Online Activity	Description of Online Tool	Good for
  Whiteboard	Whiteboard	Offers space for brainstorming and group activities.	− Instruction. − Group work. − Brainstorming. − Refocused teaching. − Icebreakers.
  Group Discussion	Breakout Rooms	Allows for multiple, simultaneous, small group interactions that are separate from the main group.	− Group work. − Laboratory work.

  ^a This table provides examples and is neither detailed nor exhaustive.

  .
• Learn about communication techniques that are effective in an online environment; this training may be provided by TLS.

Course Structure (Concurrently with Educator Preparation)

• Formulate the major topics that the course will cover. These need to be congruent with other courses in a curriculum/program.
• Formulate the Course Learning Outcomes (CLOs) for the course that are congruent with the course topics and align with student objectives (or student outcomes) of an educational program. TLS can be charged with training educators in formulating informative CLOs.

Course Material

• Design the course material, that is, the content of each class session/module. Active-learning activities are designed concurrently with, and are incorporated within, the material that is being taught.
• Design the “direct assessments” (for example, exams, quizzes, projects, presentations) concurrently with the course material; these direct assessments are linked to CLOs. There should be reciprocal links between ‘what and how’ material is delivered with ‘what and how’ the material is assessed: just as the taught material dictate assessments, assessments should inform the material that is taught. Indeed, such a system links taught material to CLOs, since the achievement and attainment of CLOs is primarily measured from direct assessments.
• As the course material is being prepared, educators may need to refine their “Course Structure” and or undergo further “Educator Preparation”.

There are potential benefits and drawbacks in carrying out online direct assessments; it is worth mentioning that some online direct assessments can also be included in traditional in-class delivery modes through the LMS.

Some of the potential benefits are:

• Educators can more easily design more frequent assessments that allow them to get more recurring feedback on student performance; this also imposes some regulation on study habits of students [60].
• Students receive immediate feedback on their performance on direct assessments, if automatically graded, for example for assessments that rely on choice formats (for example, multiple choice or true/false).
• Students may acquire some flexibility in their timeframes to complete some assessments.
• Students may experience reduced anxiety due to the change of venue from a traditional classroom [61].
• Students may demonstrate improved performance and satisfaction in the course [62,63,64,65].

Some of the potential drawbacks are:

• Educators may rely more heavily on choice formats than other forms, like short-answers and active drawing, that could be more reliable indicators of CLO acquisition.
• Academic dishonesty may become more prevalent [66].
• Mechanisms that are designed to lessen the likelihood of academic dishonesty may infringe on privacy.
• There may be a possible disconnect between test scores and cognitive engagement [67].

It is clear that online direct assessments have many challenges and provide some opportunities. As such, institutions need to establish clear guidelines that educators can use for their online and traditional forms of direct assessments [68,69,70].

Lesson Planner

A high-level outcome-based lesson planner can assist in the design of courses with virtual, real-time modes of delivery. This planner can be adapted for the design of class sessions (a grouping of activities traditionally referred to as “lectures”) within a course and includes the following elements (

Table 2. High-level outcome-based lesson planner.

Topics	Sessions	Activities: Type of Demonstration, Instruction, Discussion, or Other Activity	Delivery Method: How Activity/Content Will Be Delivered?	Time	Assessment Components a
Topic 1	Session 1 CLOs 1, 2, and 3	a. Presenting	a. PowerPoint slides	a. 10–15 min	Exam 1- Question 1 (CLOs 1, 3) b
		b. Demonstrating code	b. Online compiler (through shared screen)	b. 10–15 min
		c. Demonstrating concept + group discussion	c. Online video + breakout rooms (See Table 1)	c. 5 min	Quiz 1- Question 1 (CLO 2) b
	Session 2 CLOs x, y, and z c	TBD c	TBD c	TBD c	TBD c

^a Parts of assessment instruments are listed in this column. Each part assesses Course Learning Outcomes (CLOs) that had been drafted by the educator of that course. ^b These are examples of parts of assessment instruments that assess topics and specific CLOs covered in Session 1. For example, Question 1 in Exam 1 may assess students’ understanding of “coding”, which assesses CLOs 1 and 3 of the course; Exam 1 may contain additional questions from other sessions (or a synthesis from multiple). ^c The specifics would be inserted by the educator of the course.

):

• Each course topic is divided into a series of sessions.
• Each session has activities that incorporate active learning practices.
• Each activity is designated a delivery method in the context of a virtual, real-time class.
• Each delivery method is associated with an estimated time for completion.
• Direct assessment components are linked to CLOs and aligned with activities.

General Considerations during the Design of Class Sessions

Educators should be mindful that students may feel isolated, uncertain, and distracted in a virtual learning environment. Educators can alleviate this reality by planning lessons that set expectations and use the platform tools to create dialogue and invite social connections. Educators should foster student success by providing required information and resources, planning a soft opening for each session, and incorporating frequent interactions. General considerations for planning class sessions in a virtual environment include:

• How much setup is needed by the educator?
• How much instruction is needed for students to engage in the tasks? This time should be included in the planning (Table 2).
• Are there possible accessibility issues for some students?
• Can some activities be accomplished asynchronously, for example, as an assignment (such as viewing a video) that students can complete at their own pace before a re-al-time session?
• Can some material, especially those that may be impacted by student accessibility, be flipped such that class sessions are more devoted to interactions between/among educators and students.

Common delivery methods in a virtual, real-time class include clip arts, audio, video, animation, drawings, and/or other content. The following are some technical considerations when considering real-time delivery methods:

• Presentations: Weigh between uploading an application and onscreen sharing of the application.
• Animations: Weigh between using PowerPoint slide animations and building animations with multiple slides.
• Other Factors: Additional factors may impact the quality of the content delivery and the students’ accessibility. These include: Web Browser vs. Application vs. Mobile Application; network throughput for videos; access to outside sources; firewall settings and Virtual Private Network (VPN) since they cause major bandwidth choke; and audio quality of students.

The planned time component in Table 2 helps educators (E) deliver all the planned material, especially in a virtual learning environment that incorporates active learning practices. This time is refined during the rehearsal of the script.

4.1.2. Rehearse Script and Timing

The landscape of virtual, real-time course delivery is uncharted or unexplored territory for most educators. To engage students during the online delivery of a lesson, educators should practice delivery at least one day before the scheduled date utilizing all the designed material and intended delivery tools; this allows educators to adjust the allotted time for each activity (Table 2). Technical issue(s), should they arise and are not resolved by the Educator (E), should be reported to the ICTS (I in Figure 2). Should these issue(s) persist, the educator should adjust content and/or delivery of the session.

4.1.3. Finalize Files and Upload to Course LMS

After rehearsing a class session, it is likely that educators may adjust some of the content, which should be updated in the LMS. As a backup measure, a copy of the class/session content should be placed in a public cloud storage that is accessible by students.

4.1.4. Prepare a Contingency Plan

Many events can affect the flow of a virtual, real-time lesson. educators should prepare contingency plans, with the assistance of ICTS, should things go wrong during the virtual delivery of a lesson.

Table 3. Contingency plans for sample events.

Event	Contingency Plan
Students cannot log in.	Ensure ICTS is available to provide immediate support.
Educator audio is not working.	Log onto a second machine, and/or have another smart device on standby.
Electricity goes out for educator.	Ensure that another smart device is fully charged and internet router has alternative power.
Video will not play.	Have videos available outside of the platform on a separate server (for example, YouTube). Log onto a second machine, and/or have another smart device on standby.
The student presenter disappears.	Have a copy of the student presenters’ slide open on your machine. Have a backup presenter(s).
Educator is late.	Provide students with contact information of the educator.
Participants cannot use chat.	Have an alternative chat access, like WhatsApp.
Application Sharing suddenly stops.	Know the tools and be flexible to spontaneously adjust during the live session. Plan alternative activities.
Session did not record.	Use a screen recording application with external audio as a backup plan.
Slides will not load.	Have a copy of your slides in a PDF format
Demonstration site is down.	Have a static copy of the demonstration site stored on a public drive that is accessible to students.
Pre-work was not sent.	Preload any document onto one or shared public drive with a link to access documents

shows samples of different events that can interrupt a virtual lesson and possible contingency plans.

4.2. Module 2—Online Lesson Delivery

The online lesson delivery module has three phases (Figure 2). These are outlined below.

4.2.1. Module 2/Phase 1: Ensure Delivery Tools Are Operational

It is common that educators encounter unforeseen problems with some of the technology. Educators should therefore test all the tools scheduled for a class session one hour before the scheduled delivery. Any problems should be immediately communicated to ICTS, ideally from within the LMS.

The audio/video are essential virtual delivery tools and must always be tested to ensure quality lesson delivery. The following general audio/video guidelines are recommended and can be incorporated in the training provided by TLS to educators in the first module (Figure 2 and Figure 3):

• Audio: The sound of a speakers’ voice matters as much as the content of the message, especially in a virtual learning environment. Therefore, ensure crystal-clear audio and make the most out of the voice’s volume, pitch, tone, breadth, and rate of speech.
• Video: Use live video to help create a sense of community anytime there’s available bandwidth. Ensure the following for optimal effect:
  • Put light in front of you.
  • Place camera at eye level.
  • Keep an appropriate distance from the device camera (use palm technique for measurement).
  • Be aware of your background.

4.2.2. Module 2/Phase 2: Record Class Session

Recordings of class sessions, preferably posted on the LMS, facilitates the review by students of the content and activities of class sessions. This recording of class sessions is even more crucial in virtual, real-time classes that are susceptible to technical interruptions that may impede some students from attending all, or segments of, class sessions; such recordings become their primary means of learning the class material. In keeping with the European Union, General Data Protection Regulation (GDPR), and similar regulations in other jurisdictions, the consent of all participants in class sessions must be acquired before recording begins.

4.2.3. Module 2/Phase 3: Welcome Students and Conduct Activities

Verbal and visual communication skills are critical during a virtual, real-time classroom session. The educators’ voices, presentation styles, and online presence are key factors in online teaching and learning. The online presence of educators is analogous to their physical presence in a classroom; as such, educators should be adept at using all the tools in an online environment, some of which are shown in

Table 4. Online skills for educators in a virtual class environment.

Task
Conduct audio checks.	Monitor chat.
Upload slides to present.	Point and click.
Organize participants into breakout groups.	Locate and paste URL.
Turn on/off enhanced participant rights.	Set up activities.
Provide instructions.	Facilitate discussion.
Transfer handout file.	Identify open microphones and mute them.
Clear status indicators.	Respond to technical questions.

;

Table 5. Communication Tips During Course Delivery.

Communication Tips
Do not assume the audience is comfortable learning online; provide resources such as Quick Tip Cards before the session. This will make participants feel more at ease and able to focus on content.	Do not be afraid of the silence. Silence does not necessarily mean participants are disengaged; they just might be processing. Allow time for them to answer and react (TIP: Silently count to 10 before advancing). Be careful of proceeding before they have had enough time.
Do not read from the screen: participants can read for themselves. Your voice track should supplement content on the screen.	Do not apologize for the tool. The virtual experience is not the same as in-person, but you can accomplish the same objectives and do almost all of the same activities.
Avoid the use of filler words like “uh,” “um,” “like,” “because,” or “you know?”: Record yourself during a dry run and determine what your fillers are; then practice them out of you.	Be careful of “rambling”—nerves often cause this. If you start going too fast or get off track, just stop, mute your audio, take a deep breath, un-mute the audio, and begin again.
Set the expectation at the beginning on how they are to engage, ask questions, or provide feedback.	Keep your tone conversational, but still sound professional through word-choice.
Modulate and project your voice; avoid being monotone or mumbling.	Don’t wait until the end to ask for questions: engage participants early and often.
Give clear and succinct instructions when directing activities.

and

Table 6. Virtual classroom presentation: best practices.

If You Say in the Physical Classroom	Then You Would Say in the Virtual Classroom	Virtual Class Collaboration Feature to Use
“Let me demonstrate…”	“Please look at my screen as I start my application sharing.”	Application Sharing
“Select this or that …”	“Please respond by selecting green check or red X found under your Status Indicators.”	Status Indicator Icons
“Explore this website on your own.”	“Please click the link provided in the Chat area. This will open a new browser window.”	Hyperlink text in Chat or Application; Share Web Content
“Share an example of your own.”	“Please click the Raise Hand icon to indicate if you would like to share.” “Please post your example in the Chat area.”	VOIP—Pass microphone privileges; Status Indicators; Chat
“Draw on the flip chart to…”	“Using your annotation tools, found on the top left corner of your screen, click your Ellipse tool and then click on the screen to …”	Annotation Tools
“Please fill out the evaluation form.”	“Please complete the assessment now being opened on your screen.”	Testing or Polling
“Take a 10-min break.”	“We will now take a 10 min break. Let me know you have returned by giving me a Green check.”	Status Indicator
“I see that you are Confused.”	“How did that work for you? Please raise your hand or post in the Chat area.”	Status Indicator; Chat
“Turn to the practice unit that begins on page …”	“Download the assignment.”	Materials Panel; Share Pod; Push material to participants
“We will now watch a video.”	“I will now play a video. The video should now be displaying.	Video File; Hyperlink text to a video
“Let’s brainstorm.”	“Type your responses in Chat…” “Click your Raise Hand icon if you would like to share…”	Chat Text; Status Indicators; Whiteboard

provide communication tips during course delivery. These technological and communication practices can be incorporated in the training by ICTS and TLS in the first module (Figure 2 and Figure 3).

There are platforms that can be utilized to engage students. For example, Kahoot supports learning and adds active participation in the classroom through gamification; this improves the dynamics of the online sessions by incorporating concept exploration, implementing active learning practices, and increasing comprehension and motivation [71]. Though in its emergent stages, gamification has been successfully used in online sessions to motivate students by creating an engaging learning experience that keep students focused on the learning task and its application [72]. Using platforms like Kahoot provides educators with immediate feedback as they test students on skills they have learned, thus allowing educators to adjust their course design and delivery accordingly.

4.3. Module 3—Indirect Assessment of Educators

The purpose of this module is to solicit feedback from peers of educators and students on the performance of educators and/or on the students’ perceived acquisition of knowledge and skills. The feedback from these indirect assessments, while more based on perception, complement the results from the direct assessment that are linked to the CLOs, as outlined above. The indirect assessment of educators, and feedback, may be more important in virtual, real-time courses because the online platform reduces personal interactions that facilitate the evaluation of understanding by students. There are typically three kinds of indirect assessments:

A.

Peer Assessment: This assessment of the performance of educators is usually carried out by colleagues of educators and/or personnel involved in the TLS.

B.

Student Course Feedback: This feedback gauges students’ perception of issues, including course design, course delivery, and educator performance.

C.

Student Self-Assessment of CLOs: This feedback is designed to assess students’ self-perception of their performance on CLOs.

This module contains three phases (Figure 2).

4.3.1. Module 3/Phase 1: Design Rubrics for Indirect Assessments

Rubrics should be designed for each type of indirect assessment. Ideally, these rubrics are deployed as forms or surveys that are deployed from within the LMS:

A.

Peer Assessment: A generic peer assessment rubric that evaluates different aspects of course design and delivery can be generated by TLS. Educators and their colleagues may engage in discussions on which elements of the rubric would be assessed by the colleague in a class session, depending on the activities planned in that session.

B.

Student Course Feedback: A generic student course feedback rubric can be generated by TLS. Educators, including their academic units, may choose from, or add to, elements of this rubric for the evaluation of their courses.

C.

Student Self-Assessment of CLOs: These indirect assessment of learning outcomes surveys, also known as indirect assessment instruments in outcome-based assessments, may be designed by educators with the assistance of TLS. These surveys gauge the students’ assessment of their own attainment/achievement of the CLOs. It would be ideal, though hard to implement, to get this feedback from students after every virtual session as this provides educators with information on what to reinforce in future sessions, assignments, and activities. It is recommended that such surveys are deployed regularly, perhaps at the conclusion of each course topic.

4.3.2. Module 3/Phase 2: Deploy Indirect Assessments

ICTS could help in establishing support systems for the deployment and collection of in-direct assessments.

A.

Peer Assessment: These are done at least once in a course, typically during the middle of the semester. The assessors prepare their reports, which are immediately dis-cussed with educators, thus giving time for educators to adjust the design and/or delivery of their current courses.

B.

Student Course Feedback: These are typically deployed at the end of the course and could aid, when combined with results from direct assessments, in ameliorating the performance of the educator, and/or the design and/or delivery of the course in the future.

C.

Student Self-Assessment of CLOs: this survey is ideally deployed several times in a course to acquire continuous feedback. If deployed frequently during the course, educators may consider mechanisms for ensuring the timely completion of the surveys by students. For example, educators may restrict access by students to future topics until they have submitted the survey; alternatively, educators may consider assigning a grade weight to the surveys.

4.3.3. Module 3/Phase 3: Generate Action Plans

Based on the direct and indirect assessment results, educators/academic units can adjust course content, activities, and/or delivery methods, in the current and/or co-/pre-requisite courses to bolster low, collective (class-wide) performance or feedback. These assessments could also guide the self-development of educators, for example by steering them towards specific trainings by TLS or ICTS. While not the focus of this article, institutions should also have additional mechanisms to assist struggling students, relative to the class mean.

This QA framework therefore incorporates both direct and indirect assessments as feedback mechanisms for the continuous improvement of courses.

5. Discussion

This work proposes a generic QA framework for the design and delivery of virtual, real-time courses that are compliant with outcome-based accreditation bodies. It comprises three constituents: Information and Communication Technology Support (ICTS), Teaching and Learning Support (TLS), and Course Management System (CMS). This generic QA framework provides the organizational structure of the teaching and learning landscape; it can be adjusted based on the capabilities and goals of an institution. For example, the TLS constituent may be a formal, funded unit in an institution, for example a Teaching and Learning Center; alternatively, it can engender the training of a few educators who would provide this service to the rest of the educators. A QA framework defines the teaching and learning goals and operational structure. This framework is therefore an essential component of an institutional QA System that guides the design of policies and procedures and ensures continuous improvement of operations and of the QA framework itself.

To ensure continuous improvement, both at the macro level (policy, procedures, and QA framework design), and the micro level (implementation of procedures by academic units), a QA system that incorporates a continuous improving cycle with proper assessment and feedback is required (Figure 4). The continuous improvement cycle of the QA system that is outlined below can be used by most educational institutions, albeit with some adjustments. In the context of QA, these adjustments may be related to the culture and mission of the educational institution. The QA system comprises four modules that are briefly discussed below in the context of the QA framework.

5.1. Educational Institution Macro Level

5.1.1. Module—Evaluate and (Re-)Design QA Framework

A QA framework is an essential tool for starting institutions since it guides the development of educational policies and procedures; it can also be an effective tool for established institutions who wish to refine their educational goals and adjust their policies and procedures. However, the QA framework itself needs to be systematically adjusted and improved. The feedback received from the “Assess Effectiveness of Implementation” module in the QA System is used to evaluate the effectiveness of the framework. This evaluation may result in the amelioration of the QA framework and/or its associated policies and procedures.

5.1.2. Module—Design Policies and Procedures Based on QA Framework

The QA framework guides the development of policies (laws) and procedures (implementation details) that regulate and guide the implementation of the QA framework. Indeed, the policies and procedures of the Course Management System (CMS), Teaching and Learning Support (TLS), and the Information and Communication Technology Support (ICTS) constituents must be coordinately developed and implemented to ensure the overall operational success. In addition, institutions should be aware of the requirements of their targeted accreditation bodies when constructing QA frameworks and their associated policies and procedures. Procedures provide detailed instructions with forms, templates, and checklists on how to accomplish rules established in policies.

5.2. Educational Institution Micro Level

5.2.1. Module—Implement Procedures by Academic/Non-Academic Units

The established procedures are used by institutional educators and units (formal entities, such as offices, centers, departments, and/or faculties) to engage students; feedback from educators, students, and units should be used to evaluate the feasibility of the implementation plan and can be mechanisms for strategic institutional improvement. For example, some active learning methods may not be implementable in the current LMS, so the LMS capabilities may be upgraded.

5.2.2. Module—Assess Effectiveness of Implementation

To assess the educational experience, the QA System requires operational assessment instruments that evaluate compliance with policies and procedures which were designed based on the QA framework; for example, operational assessment instruments could be incorporated in peer assessments and/or student course feedback. These instruments determine whether the QA framework adopted by the institution can/is functioning effectively. Furthermore, the direct and indirect course assessments outlined in this work are utilized to gauge student and educator performance and development; these course assessments are used at the Micro level to improve the educational experience, for example using feedback mechanisms outlined in this work (see Figure 2 and Figure 3). Data from the operational and the course assessment instruments could be used to adjust the QA framework and/or its associated policies and procedures.

The complete QA framework with its well-developed constituents should clearly delineate its stakeholders. Indeed, this article refers to educators as the primary stakeholders in all aspects of the CMS constituent. While several of the course activities rely primarily on educators responsible for the courses, the educators’ academic units (for example, departments) should be integral to managing programs/curricula and interactions/accountability with other constituents (TLS and ICTS). Thus, feedback from assessments can be used to ameliorate course design, course delivery, curricula, and/or the QA framework and its associated policies and procedures.

6. Conclusions

The full or partial shift of course delivery to the virtual space in real-time is challenging, especially if there is a desire to maintain a learner-centric educational discourse. The QA Framework introduced in this work can serve as a starting point for institutions to establish new, or amend existing, policies and procedures; the proposed framework itself would first have to be adjusted to be compatible with the missions of institutions and their available resources.

Institutions have their own missions, modes of operations, infrastructures, facilities, and budgets. All of these are factors that may influence the structures of their QA frameworks. For example, budgetary constraints may require managing, adapting, or innovating active learning practices that can be implemented using technologies that are already available in the institution. Institutions that are being founded may decide to first generate a QA framework and subsequently establish some of their formal entities on its basis. For example, the TLS may be an “Office of Teaching and Learning” whose duties include the ones listed in their QA framework; similarly, the ICTS may be an “Office of Information and Communication Technology”. Established institutions may use a QA framework to review the effectiveness of their existing educational operations and correspondingly establish new formal entities or assign new duties to existing ones.

Just as this work has focused on the “Course Management System” constituent of the QA framework, future work will develop the modules in the other two constituents: “Teaching and Learning Support” and “Information and Communication Technology Support”. The QA framework will then be used to develop other components of the QA system; this system includes continuous improvement mechanisms that are necessary for institutions to adapt to the constantly evolving educational and technological landscapes.