A Systematic Literature Review on the Quality of MOOCs

Abstract

This paper presents the findings from a systematic literature review on the quality of massive open online courses (MOOCs). The main research question was “How can the quality criteria for MOOCs identified in the analysed studies from the systematic literature review be best organised in a categorisation scheme?” The systematic literature review was conducted using the PRISMA procedures. After conducting the screening and eligibility analysis according the pre-defined criteria, 103 studies were finally selected. The analysis was done in iterative cycles for continuous improvements of the assignments and clustering of the quality criteria. The final version was validated in consensus through the categorisation and assignment of all 103 studies in a consistent way to four dimensions (pedagogical, organisational, technological, and social) and their sub-categories. This quality framework can be re-used in future MOOC research and the discussion of the analysed studies provides a current literature overview on the quality of MOOCs.

1. Introduction

Currently, innovative education is often wrongly characterised as involving continual introduction of new technologies [1,2]. Thus, to understand innovative education, it is necessary to re-visit the social function of pedagogy in the development and growth of individual learners. In the European political context, some educational systems have been adopting an increasingly open approach, in line with the global movement for open education and open educational resources (OER). The OER acronym was first adopted at the 2002 UNESCO forum on the impact of open educational content for higher education in developing countries, funded by the William and Flora Hewlett Foundation [3]. The OER movement can be considered one of the most significant effects of digital innovation in the educational field [1]. The idea behind this movement is to make all created educational resources openly and freely available through appropriate repositories [4]. This allows all learners worldwide to re-use, adapt, and take advantage of these learning opportunities, in order to enable education as a common good [5].

During recent years, massive open online courses (MOOCs) became popular, taking the form of free online courses that were accessible to everybody and in which a large number of learners enrolled [6]. To date, the number of MOOCs, learners, and providers is still increasing continually: Despite blame as a buzzword and hype already in the year 2012, the number of offered MOOCs as well as of learners is growing today, with the main focus and popularity on educational research [7,8]. Due to its unbroken growth, this special type of OER has prompted a lot of research [9,10,11]. MOOCs prompted many authors to investigate the potential of such a so-called “disruptive innovation” [12,13,14,15], even though this innovation is not disruptive, but an adaptation of old and well-established pedagogical approaches in online learning [16]. Ref. [17] defined a MOOC as a didactic product that guarantees a structured learning path, including a syllabus and explicit didactic objectives, materials, and learning support activities, and an evaluation system based on quizzes, exercises or projects, and a certification process. A MOOC has to be usable through an online platform and must be designed and managed in such a way as to guarantee its scalability, so that it can be enjoyed by a large number of learners. In addition, it has to be accessible to all without prerequisites for registration and participation.

Different types of MOOCs and different pedagogical models for MOOCs were developed, including the initial distinction between cMOOCs and xMOOCs. Within that distinction, cMOOCs are mainly designed to support processes of aggregation, relationships, creation, and the sharing of knowledge, which is also known and promoted as a collaborative or connectivist approach [18] (for a critical debate on connectivism, see [19,20]). In contrast, xMOOCs are more institutionalised and rigorously structured, focused on cultural reproduction through videos and quizzes, known as a behaviourist approach [10]. However, research in the last few years has shown that such a separation and distinction cannot be made, due to the increasing overlap and mix of these two extremes [11,20,21]. In order to understand whether MOOCs can actually represent a revolutionary innovation in the educational field, it is necessary to investigate how they have been conceived, developed, and implemented and to analyse their pedagogical quality, learning effects, and impact [22,23,24,25]. As a starting point, the quality of MOOCs is understood as multi-dimensional and not definable in one single phrase; it has to reflect the different views, needs, and interests of the learners and other stakeholders, with specific focus on the pedagogical aspects that are most relevant for the quality of MOOCs, as highlighted in the latest research [1,10,17,20,24,26,27,28,29].

It is complex to set general quality standards for MOOCs, as there are many types of MOOCs, as well as many purposes of MOOC providers, who can offer courses to disseminate knowledge, to attract new students, or to increase learners’ motivation [30,31]. According to many authors, attention has to focus on the process rather than on the outcome, since the MOOC should be seen as a means of discovery and a way of experiencing for the learners. Therefore, current rankings and rating systems for their quality (such as simple five-star scales) are failing [20,21,26,30,31,32,33,34]. Ref. [35] promoted MOOCs as disruptive technology, and she proposed different strategies for enhancing the learner experience and quality of MOOCs. The success or failure of a MOOC depends on how specific quality criteria are met, which has also been confirmed by the latest MOOC research, including the findings from the Global MOOC Quality Survey [21,27,28]. Refs. [2,25] pointed out that MOOC research requires taking a new stance towards quality improvement. Many authors have also asked whether high drop-out rates should be interpreted as an indicator of (poor) quality or whether a high drop-out rate is an indicator of different preferences and effective choices by the learners [36]. According to [37], analysis of the motivations that push learners to leave the MOOC could improve the teaching quality of MOOCs in general. Ref. [38], on the other hand, considered that the non-completion rate does not determine the quality of a MOOC. If we focus only on quantitative comparison of the enrolment rate and the completion rate for the course, we lose sight of other elements that should be considered. First of all, there can be diverse reasons why learners did not complete the MOOC: We must distinguish between those learners who did not intend to try to complete the didactic activities (but are labelled as drop-outs) and those drop-outs who tried to complete the activities but did not meet the criteria [39]. These data are difficult to obtain, since MOOCs do not easily allow this type of monitoring [40].

Beyond the obvious difficulties in mapping thousands of learners, there are different ways for learners to participate in and to benefit from MOOCs, even without certification [41,42]. Therefore, besides the drop-out rates, several other characteristics need to be considered as far as MOOC quality [23]. They include, for example, the impact of the different evaluation approaches adopted, the differences emerging in the implementation phase, the choice of pedagogical approach, and the opportunity to re-design the MOOCs to include additional alternative pathways and materials for learners who could not pass tests. For [35], instructional design should be instrumental in encouraging reflection, facilitating dialogue and collaboration, applying theory to practice, creating peer communities, encouraging creativity, and motivating learners. Research conducted by [24] highlighted the poor teaching quality of MOOCs, in which the principles of instructional design that the authors used for the analysis of the selected MOOCs were rarely respected, resulting in a larger focus on other aspects related purely to the organisation and presentation of the teaching materials.

Do MOOCs risk being a “colorful repetition” [43] of traditional individual learning and represent only cultural reproduction? Although MOOCs represent a new opportunity for higher education, this remains a very important issue to consider. To give an example, there could be significant obstacles to implementing a pedagogical model that intends to enhance some basic principles of social constructivism, as most MOOCs are based on what Horton defined as the logic of “WAVWAVWAVAAQ” (as cited in [44]: Watch a Video Watch a Video Watch a Video And Attempt a Quiz.

A first systematic literature review covered the MOOC research studies published up to the year 2012 [45]. Afterwards, none of the available systematic literature reviews provided a full overview with a focus on the quality of MOOCs: [9] presented an analysis of the data about MOOCs (up to the year 2013), Ref. [46] offered a qualitative analysis of how MOOCs support academic employability (up to the year 2015), and three systematic literature reviews focused their studies up to the year 2018 on selected specific topics—[47] on engagement and retention in vocational education and training MOOCs, [48] on the accessibility of MOOCs, and [49] on research methods and topics in empirical MOOCs. Starting with this complex situation, the authors conducted a systematic literature review on the quality of MOOCs in order to collect data about the different studies carried out by researchers during the last seven years (2013 to 2019) and their findings and to use those data in order to develop a framework within which to organise the quality categories and criteria of MOOCs.

2. Materials and Methods

This article provides the results from a systematic literature review study that followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) Statement and its procedures [50]. The research question for our systematic literature review was:

• How can the quality criteria for MOOCs proposed in the analysed studies from the systematic literature review be best organised in a categorisation scheme?

Systematic literature reviews and meta-analyses have become increasingly important and useful in the educational field, as there is a lack of careful validation and review of studies in educational research [51]. A systematic literature review ensures a rigorous and accountable approach to the search for, selection, and critical analysis of studies [52,53]. Systematic reviews summarise the state of the research on a topic using methods that are pre-planned and documented in a systematic review protocol. This type of review is based on the rigorous analysis of the evidence that arises from a careful evaluation of the available literature according to shared criteria. Conducting a systematic review assumes mastery of a complex work methodology. In particular, it requires a well-structured process that defines the question of the review, as well as which studies will be located, evaluated, selected, and then synthesised. To meet and fulfil all these requirements, we chose to use the methodology outlined in the PRISMA statement.

2.1. Systematic Review Process: The Methodology in the PRISMA Statement

The methodology in the PRISMA Statement [50] consists of a 27-item checklist that concerns methods, results, discussion, and funding and a four-phase flow diagram that concerns record identification and screening, study eligibility, and studies included. We adhered to all 27 items on the PRISMA checklist.

Our review takes into account literature published from January 2013 until December 2019. We decided in advance to limit our systematic review to the most recent literature from the last seven years, both to avoid having too many studies included and because previous systematic literature reviews on the quality of MOOCs have covered the time period up to 2013 [9,45]. None of the latest systematic literature reviews on MOOCs [11,21,46,47,48,49] have targeted this specific topic of the quality of MOOCs.

Two inclusion criteria were defined before the first screening, according to formal aspects: first, only studies in English and second, only studies published in journals or peer-reviewed proceedings of scientific conferences. Next, a manual check for duplicates was defined before it took place. The abstracts of the remaining studies had to be analysed in a second screening for their eligibility according to their relation and relevance to our research question. The exclusion criterion that was defined for the second screening was relevance, meaning, “Does this study have any relevance for our research question?” This exclusion criterion is very vague; therefore, it should only be used to remove studies that are obviously addressing different topics and disciplines. Where there would be any doubts, the studies should not be excluded but rather included, and kept for the following full-text analysis. The final inclusion criterion defined for that analysis was the availability of the full texts. The remaining studies had to undergo quantitative and qualitative synthesis. The results from the quantitative analysis are presented in the Results section below, whereas the Discussion section discusses the results from the qualitative analysis.

2.2. Systematic Review Process: The Realisation of the PRISMA Statement Procedures

Following the PRISMA Statement procedures as presented in the flow chart below (see Figure 1), the first phase regarding record identification was carried out by searching for studies in four electronic databases: Web of Science (Clarivate Analytics), Scopus (Elsevier), Scholar (Google database), and AlmaStart (University of Bologna database, EBSCO host).

First, the broad search term “mooc AND quality” was tested in the four databases, resulting in 22,818 records in total. This large number of records could not be analysed in an appropriate timeframe by the authors. Therefore, the authors decided to use the more restrictive search term combination “mooc AND quality AND pedagog*”, resulting in 749 records. That was considered to be a feasible number of records for a systematic literature review. The search term amendment “AND pedagog*” was selected due to our main interest in the pedagogical aspects of the MOOC quality. Starting with these 749 records, the authors followed the pre-defined PRISMA Statement procedures as described above. First, the formal screening related to non-English studies (-154) and non-peer-reviewed journal and scientific conference publications (-221) left 374 records. Then, 123 duplicates were removed. The remaining 251 records were assessed for eligibility by the pre-defined selection criteria, analysing their titles, keywords, and abstracts (-134), leaving 117 records. These 117 records were treated as collected studies for full-text evaluation. A total of 14 studies could not be found as full text and had to be removed, leading to a final total of 103 selected studies (see the list in Appendix A).

Figure 1 below presents the flow chart for the PRISMA Statement procedures and the results from our selection of studies for our systematic literature review following the PRISMA methodology.

The datasets of our systematic literature review are publicly available in the annexes to allow other researchers to re-use them and benefit from their collection and data.

3. Results

A total of 103 studies were included in the quantitative and qualitative analysis after the selection process for our systematic literature review, as described above (see Appendix A with the full list of all 103 selected studies and their coding, consisting of three letters and two digits for the publication year and used in the following). This section presents the findings from the quantitative analysis of all 103 selected studies.

The 103 studies were published in various journals and proceedings. Only one publication was the source of four included studies: International Journal for Educational Technology in Higher Education (Cos18, Dan15, Fid16, Fre18). Five publications were the source of three included studies each: the journals Educational Media International (Con15, Liu15, Oss15), European Journal of Open, Distance and E-Learning (Liy19, Pet16, Rol15), Revista Española de Pedagogía (Dua17, Lem19, Vaz17), International Journal of Emerging Technologies in Learning (Faq18, Kha15, Tah17), and the ECEL proceedings (Bar15, Fit14, Mas15). Four publications were the source of two studies each (International Review of Research in Open and Distributed Learning with Fer17, Naj15; Journal of Online Learning and Teaching with Bal14, Mil15; the ICALT proceedings with Lop15, You14; and the EMOOCs proceedings with Alo19, VdP19). The 76 other studies were all from different publications. Out of the 103 studies, one study was from 2013, 13 studies were from 2014, 24 studies were from 2015, 18 studies were from 2016, 17 studies were from 2017, 12 studies were from 2018, and 18 were studies from 2019.

The authors developed and repeatedly applied and improved a categorisation scheme for the quality criteria of MOOCs with four main dimensions: the organisational, technical, social, and pedagogical aspects of MOOCs, as proposed by [54]. The categorisation scheme was initially designed based on the results from our analysis of the previous systematic literature reviews on MOOC research [9,11,21,45], from the Global MOOC Quality Survey, and from own experiences in the design, realisation, and evaluation of MOOCs. Furthermore, in order to assure conciseness, robustness, completeness, extendability, and explainability [55], the taxonomy development process followed the “three-level model” by [56], including alternative approaches (e.g., conceptual and empirical) that could be repeated in different combinations. This iterative process allowed the authors to add, split, remove, and merge categories and their hierarchical levels that composed the categorisation scheme [55].

The first draft of the categorisation scheme was designed as a concept map and included 58 categories under the four main dimensions, on five hierarchical levels (including the four dimensions). All 103 studies were initially assigned to the defined categories during the superficial eligibility and inclusion process. During their qualitative full-text analysis, the categories were continually revised and improved by the two co-authors as follows: They independently classified the studies and each developed a new individual draft for potential categories. Afterwards, they presented the two individual drafts to each other and discussed them. Based on these two drafts and the discussion, the first common draft of the categorisation scheme was developed in consensus (first iterative cycle). For repeated improvement, the studies were separated into 10 groups of 10 or 11 studies. After revisiting and analysing the studies of one group, the authors independently revised the categories and their hierarchical levels (the next additional iterative cycle). The results were again discussed, leading to a next categorisation scheme developed in consensus (the next additional iterative cycle). In the beginning, the tendency was towards having more hierarchical levels (up to six levels), but it turned out that it became difficult to differentiate so many hierarchical levels. After 10 rounds with the 10 groups of studies, there was a total of 19 iterative cycles of assignments of the studies and refinements of the categories and their levels. In the end, a final version of the categorisation scheme for the quality criteria of MOOCs was developed, with 71 categories on four levels, including the four dimensions as the highest level (three levels without the dimensions). It was validated, as all 103 studies could be assigned and included in this final version. As to be expected from the search terms used, the pedagogical dimension was more split up and differentiated than the other three dimensions, as most studies could be assigned to the pedagogical dimension.

The 103 studies (see the list in the Appendix A) were very unevenly distributed among the four dimensions and their categories (one study could be assigned to more than one category). The vast majority could be assigned to the pedagogical dimension (88 out of 103 studies), whereas fewer studies were assigned to the three other dimensions: 18 studies to the organisational dimension, 25 studies to the technical dimension, and 19 studies to the social dimension (see the

Table A1. The quality framework for MOOCs with all 103 assigned studies.

1. ORGANISATIONAL	Lop15, Mca14, Nie16, Sto16, Str17
1.1 Institutional commitment	Fit14, Fre18, Ghi16, Kin14, Mas15
1.2 Policies and licenses	AlI19, Hai19, Nyo13, Ram17
1.3 Efficiency and cost	AlI19, Dan15, Row19, VdP19, Wam18
1.4 Sustainability and society	AlI19, Ram17
2. TECHNICAL	Gam15, Lop15, Lau16, Llo16, Ram15, Yal17, You14
2.1 Provider and platform	Bab16, Bae16, Dou14, Ebn16, Goo18, Hic17, Kin14, Pil17, Sin19
2.2 User interface	Haa19, Kha15, Kim14, Sin19, You14
2.3 Video development	Hic17, Kha15, Kim14, Man19, You14
2.4 Technical interoperability	Bro14, Faq18, Kin14, Tah17
2.5 Technology for social interaction	Amo14, Pil17, You14
3. SOCIAL	Gar17
3.1 Communication	Bab16, Bar15, Bro14, Kha15, Kin14, Mil15, Naj15
3.2 Collaboration	Fid16, Gam15, Mas15, Pet16, Saz18, Tah17
3.3 Discussion	Lau16, Pet16
3.4 Networking	Amo14, Bar15, Bas14, Sto16, Vaz17
3.5 Sharing	Lau16, Pet16
4. PEDAGOGICAL	Alb18, Gam15, Lau16, Lop17, Mar14, Sto16
4.1 Instructional design	Alo19, Con15, Dua17, Fer17, Fia18, Hli16, Fal16, Lem17, Liu15, Mar15, Naj15, OhC19, Oss15, Pet16, Pil17, Ram15, You14
4.1.1 Learning objectives	Ist15, Kha15, Ram15
4.1.2 Target group	Bas14, Hai19
4.1.3 Design approaches	Amo14, Bor17
4.1.3.1 Collaborative learning	Con15, Fid16, Oss15, Pet16, Ruh15, Saz18, Tah17, Tov15
4.1.3.2 Learner-centred	Amo14, Oss15, Pil17, Ruh15
4.1.3.3 Situated learning	Tov15
4.1.3.4 Active learning	Saa16, Tov15
4.1.3.5 Adaptive learning	Ard17, Amo14, Dan15, Hem17
4.1.3.6 Learner’s role	Avs18, Bro14, Bou17, Mas15
4.1.3.7 Teacher’s role	Avs16, Avs18, Bro14, Doo19, Mas15
4.1.3.8 PBL	Mar15
4.1.3.9 GBL	Amo14, Bai14, Bro14
4.1.3.10 IBL	Bai14
4.1.3.11 Content-oriented	Nie16
4.1.3.12 Task-oriented	Pil17
4.1.3.13 Skills-oriented	Amo14
4.1.4 Course conditions	Kha15, Pil17
4.1.5 Course description	Kha15
4.1.6 Course content	Ada14, Bon17, Bor17, Con15, Ebn16, Gam15, Hic17, Ist15, Kim14, Kin14, Naj1, Oss15, Pet16, Ram15, Yal17
4.1.7 Activities	Bon17, Con15, Ebn16, Kin14, Pet16, Ram15, Yal17
4.1.8 Resource features	Dou14, Gam15, Kim14, Kin14, Liu15, Naj15, Pil17
4.1.8.1 Accessibility	Ini19, Kim14
4.1.8.2 Usability	Haa19, Kim14
4.1.9 Assessment	Bon17, Cos18, Ist15, Pil17, Tah17, You14
4.1.9.1 Peer assessment	Amo14, Eli19, Lau16, Kha15, You15
4.1.9.2 Formative assessment	Con15
4.1.9.3 Rewards	Con15, Dan15, Kha15, Naj15
4.2 Learners’ perspective	Alu16, Che19, Dua17, Fia18, Goo18, Haa19, Rui19, Str17, Ulr15
4.2.1 Relationship teacher and peer	Gre18, Mas15
4.2.2 Self-assessment	Kha15, Naj15, Pil17
4.2.3 Personal learning environment	Bar15, Coh15, Mon15
4.2.4 Motivation	Cas15, Ebn16, Gam15, Kha15
4.2.5 Learning style	Ard17, Bae16, Bal14, Bas14, Pil17, Rol15
4.2.5.1 Individualized pathways	Ruh15, Sun16, Tah17
4.2.5.2 Self-paced learning	Oss15, Sto16, Tah17
4.3 Theoretical framework	Amo14, Oss15, Sac18, Tov15
4.3.1 Connectivism	Amo14, Oss15
4.3.2 (Socio-)constructivism	Tov15
4.4 Learning processes	Gre18, Pil17
4.4.1 Feedback	Eli19, Lee16, Mar15, Naj15
4.4.2 Guidance to learner	Bai14, Gam15, Ruh15
4.4.3 Tutoring	Ram15
4.4.4 Interactivity	Bro14, Ebn16, Gam15, Liu15, Mil15
4.5 MOOC classifications	Avs18, Bro14, Con15, Fid16, Gre18, Liy19, Mar19, Sac18, Tah17
4.6 Context	Bra15, Hsu16, Mar15, Ram15
4.6.1 Formal	Bor17, Fid16, Hai19
4.6.2 Non-formal	Cin19, Fid16
4.6.3 Informal	Fid16
4.7 Evaluation	Amo14, Lee19, OhC19, Pil18, Row19, Tah15
4.7.1 Learning analytics	Amo14, Cin19, You14
4.7.2 Educational data mining	Amo14
4.7.3 Drop-out rate	Che19, Dou14

in the Appendix C for the assignment of all 103 studies to the categories of the Quality Framework for MOOCs).

Along with the presented analytic framework for the studies as the main result, another key finding was that only one study out of all 103 selected addressed three or more dimensions.

Differences also emerged concerning general studies covering the whole dimension. There was only one study focused on the whole social dimension, but several general studies that focused on the pedagogical dimension (6); in particular, there were many general studies for the organisational (5) and the technical (7) dimensions, in relation to the smaller total number of studies for these dimensions compared to the pedagogical dimension. In addition, not only was the pedagogical dimension covered by the greatest number of studies (88), but also within that dimension was the greatest number of assignments to the different sub-categories (207). The other three dimensions were covered by 52 studies addressing 14 sub-categories: 18 studies addressing four sub-categories related to the organisational dimension, 25 studies addressing five sub-categories related to the technical dimension, and 19 studies addressing five sub-categories related to the social dimension.

Due to this unbalanced result, the overview in Figure 2 below presents only the titles of the three under-represented dimensions (organisational, technical, and social); only the pedagogical dimension shows the complete version with the three levels of sub-categories (see the full version of the quality framework with all sub-categories and four levels for all four dimensions in Figure A1 in Appendix B).

In summary, the final version of the categorisation scheme, the quality framework for MOOCs, was helpful for the clustering and differentiation of all 103 studies. The categories in the quality framework and the structuring of their levels were improved and validated through repeated assignment of the studies, which led to only three categorisation levels for the pedagogical dimension (presented in Figure 2) and only one level for the other three dimensions.

4. Discussion

In this section, the authors discuss what emerged from the qualitative analysis of all 103 selected studies. This article aims to develop a new framework for analysing current studies focusing on the pedagogical quality of MOOCs. A systematic literature review using pre-defined selection criteria was conducted, in which 103 studies were found (see the list in Appendix A) and assigned to the categories of the Quality Framework for MOOCs (see Table A1 in Appendix C).

4.1. Contribution to the MOOC Research and Literature

This systematic literature review supports and contributes to the research and literature on the quality of MOOCs in three respects.

First, this article aims at the development of a framework for the categorisation and analysis of the selected studies. The categorisation scheme was improved in iterative cycles during the analysis and assignment of the studies, leading to the final version presented in Appendix B. The framework was helpful and could be used to categorise and assign all 103 studies in a consistent way. Thus, it can be re-used in future systematic literature reviews on the quality of MOOCs.

Second, this article provides an overview of the current state of the research and literature on the quality of MOOCs. The results from the quantitative analysis of the 103 studies were reported in the section above. In the following sub-sections, we will discuss the findings from the qualitative analysis separately for each of the four general dimensions: pedagogical, organisational, technical, and social.

Third, we will discuss all of the results from the systematic literature review on a general level. Therefore, the overall findings from the analysis across all four dimensions will be presented in the last sub-section. The key finding here is that only one out of all 103 studies focused on three or more dimensions concerning MOOC quality.

4.2. Discussion of the Pedagogical Dimension

The pedagogical dimension is considered to be the most important dimension related to the quality of MOOCs. Consequently, the vast majority of the studies (88 out of 103 studies) addressed pedagogical aspects of MOOCs. Seven studies provided an analysis of (a larger number of) empirical studies, pointing out that:

• Good designs can help to promote MOOC participants’ levels of engagement and allow effective pedagogical principles to benefit learners (Liu15);
• “The majority of MOOCs scored poorly on most instructional design principles” (Mar15, p. 77);
• The instructional design of MOOCs drives desired learning outcomes (Naj15);
• It is necessary “to continue placing emphasis on the pedagogical design of MOOCs, as this educational approach can have wider benefits for on-campus learning and scholarly activity, particularly research and dissemination” (Pet16, p. 128);
• Educational organisations must improve their offerings to better satisfy their learners, especially through better MOOC design processes (Ram15), and responding to the need to specifically scaffold the integration of concepts, beyond their mastery as individual concepts (Fal16); and
• Learning design principles can foster learning in MOOCs (You14).

The other 81 studies were mainly literature reviews and case studies; due to space limitations, only the most substantial publications will be discussed.

Several studies addressed learning design and the view of MOOC learners. They particularly considered the lack of instructional design (Hli16; Pil17) and the fact that learning design and learning environment are the main factors affecting learner experiences and quality (Con15; Oss15). (Str17) emphasised the need to focus on the learners’ perspective at the micro level, demanding specific learning opportunities and different learning experiences. (Alu16) considered that continual feedback from students is also crucial in creating more effective MOOCs. (Ulr15) reflected on perceptions and expectations regarding MOOCs, whereas (Che19) underlined the impact of misconceptions.

Other studies discussed the pedagogical tools and theories used in MOOCs. The qualitative study by (Tov15) explored the range of pedagogical tools used in 24 MOOCs, finding that the range of pedagogical practices currently used in MOOCs tends toward an objectivist-individual approach. (Oss15) claimed that “MOOCs rely on connectivism and co-construction of knowledge through connections and negotiation” (p. 281), whereas (Amo14) stated that “connectivism does not provide an adequate explanation of learning phenomena in Web 2.0, and therefore it is not able to provide an adequate pedagogy for MOOCs” (p. 79). (Pil17) considered the facilitation of self-directed learning processes and communities to be strategic components and demanded diverse assessment types and personalised feedback as guidance for learners.

Some studies focused on taxonomies and contexts of MOOCs going beyond the classical distinction between cMOOCs and xMOOCs. (Con15) described five different MOOC types: associative, cognitive, constructivist, situative, or connectivist. (Alb18) proposed distinguishing between the three dimensions of presage, process, and product for MOOC research, whereas (Gre18) introduced a differentiation between formal, conventional, and professional MOOCs. (Mar15) and (Ram15) considered the context as a place where learners can apply their knowledge. Research conducted by (Bra15) aimed to understand how students perceive learning outside the classroom through social media, online courses, school websites, and private tutoring. They found that more than 96% agreed or strongly agreed that learning outside the classroom has a positive impact on education, which confirms the potential of MOOCs.

Further studies related the pedagogical dimension to other fields and disciplines. (Hsu16) underlined that a nuanced understanding of the complex relationships among technology, content, and pedagogy is required for high-quality teaching. Some authors considered learning analytics as an effective way to improve the learning design and processes of MOOCs (Amo14; Tah15; You14). (AlI19) underlined that MOOCs are not disruptive innovations, but offer new learning opportunities.

Within the pedagogical dimension, it was surprising to us that many studies focused on attempts to differentiate types of MOOCs and pedagogical methodologies such as connectivism that are not proven and not even well defined (Amo14). Furthermore, most studies assigned to the pedagogical dimension did not present empirical experiments and evidence-based results (only seven out of 88 studies) but addressed theoretical proposals and presentations of potential design conceptions and guidelines: That confirms the concerns about the quality of MOOCs and related research [24,25,26,27]. It seems that studies of the pedagogical dimension are still in the beginning, surface-level stages of analysis of MOOCs. They used buzz words such as “learning analytics” and “educational data mining,” promoting their large potential without any evidence for such statements. In the meantime, the Quality Reference Framework (QRF) was developed and validated in collaboration with the MOOC community, which contains more quality criteria clustered in three dimensions and perspectives to be addressed in the design of MOOCs [28].

4.3. Discussion of the Organisational Dimension

The organisational dimension included different types of studies (18 studies in total) that focused on the organisational dimension at the general level in distinctive ways. Three studies included a theoretical analysis considering the organisational dimension to be most fundamental for assuring the quality of MOOCs (McA14; Sto15) or business revenue for educational institutions (Nie16). Two other studies analysed learning strategies and experiences at different levels, addressing the organisational dimension at the macro level (Lop15; Str17). Six other studies focused on specific organisational aspects. There were three studies related to organisational commitment to the quality of MOOCs. (Ghi16) stated in her literature review about enhancing the quality of MOOCs that the MOOC evaluation literature is still undecided between the need to adopt one of the few quality enhancement frameworks specifically created for MOOCs or to re-use the available e-learning quality models. (Kin14) defined an approach called “fit for purpose” that “produced a high quality learning experience that also supported institutional goals” (p. 117). In addition, (Mas15) and (Fit14) considered that MOOCs are potentially highly disruptive and present challenges to institutions to overcome issues connected to staffing tensions and role alterations. As for policies and licenses, (Nyo13) highlighted the importance of this category, especially for MOOC licenses that are “blurring the lines between traditional educational values and the commercial enterprise” (p. 671). As for efficiency and costs, (Dan15) wondered whether the future of MOOCs will support adaptive learning or only serve as a business model for institutions.

For the organisational dimension, it was remarkable that the development and introduction of institutional policies and strategies including visions and mission statements as well as the organisational evaluation and continuous improvement cycles were not mentioned or addressed in the studies. Thus, all studies assigned to the organisational dimension ignored research results highlighting the need for iterative cycles in organisational development to achieve quality improvements as demanded by [1,2].

4.4. Discussion of the Technical Dimension

The technical dimension was strongly addressed by 25 studies in total. Seven studies discussed the technical dimension using different criteria. (Lau16) offered a broad overview of the technical dimension considered to be strongly related to the pedagogical approaches in MOOCs, especially in order to support collaborative learning. Three other studies (Gam15; Ram15; Yal17) considered the technical dimension to be referring to a group of hardware and software tools, features, and standards that support learning processes and could even improve them. Three other studies (Lop15; Llo16; You14) took into account the technical dimension as an essential basis for MOOC challenges, such as the drop-out rate, lack of human interaction, assessment issues, and pedagogical approaches. The majority of the studies addressing this dimension considered MOOC aggregators and delivery systems as the most important technical aspects for the support of learning processes. Six studies highlighted the choice of the platform as fundamental, but for different reasons: to ensure interaction (Pil17), for decision-making (Kin14), for negotiation (Ebn16), for time management (Naw14), for collaborative learning (Bae16), or in general for MOOC quality (Bab16). The user interface and video development were discussed also in six studies, but from different points of view. (Kha15) and (You14) combined the technical implementation of interface and video in order to create an evaluation grid for observing xMOOCs to find several levels of underlying structures and design concepts in order to assure the pedagogical quality of MOOCs. The empirical study by (Kim14) considered video lectures especially, in order to analyse the learner’s interactions “depending on the visual, pedagogical, and stylistic properties of the video” (p. 8) and video player interfaces as an improvement of navigation. (Tah17) considered interoperability as a primary need: “To meet this challenge, standards have been proposed to assist interaction flow and implementation of the differentiation process” (p. 210). In addition, (Bro14) stated that a virtual learning environment (VLE/LMS) has to be enhanced with social features or combined with a community or network-like environment to foster relationships and interactions. (Kin14) considered technical design decisions to be important for optimising accessibility and scaffolding. Other authors considered technology strategic for supporting social interactions “in order to convert MOOCs into effective and adaptive learning environment” (Amo14, p. 84), an important way to achieve collaboration in MOOCs (You14), and a way to encourage learners’ participation and engagement (Pil17).

For the technical dimension, it was striking that the challenges involved in offering learning at scale for massive audiences and related technical issues for the online environment and its support were not covered by the studies, in contrast to the results by [22]. In addition, the majority of the studies assigned to the technical dimension did not reflect on the implications and complexity of MOOC integration into existing environments, as indicated by [52,54].

4.5. Discussion of the Social Dimension

The social dimension was addressed a little less often, in a total of 19 studies. (Gar17) pointed out the need for collective intelligence to develop a sense of community: When students learn together intentionally and informally in networked online environments, they form small and temporary communities. The communication aspects included building community and supporting engagement (Bar15; Kin14; Mil15); supporting instructional presence, interaction, and participation (Bro14; Kha15; Naj15); and creating a personal learning environment (Bab16). However, whereas communication referred mostly to aspects of learners’ interactions, collaboration was more related to the pedagogical model adopted. In particular, the empirical studies by (Fid16) and (Gam15) considered collaboration to be a pedagogical resource that allows for the measurement of the achievement and effectiveness of MOOCs. Findings from these studies showed that the number of drop-outs decreases when the cooperation level increases. Furthermore, collaboration is not confined to shared resources—in fact, the creation of knowledge sharing underlies the collaborative strategy. Three other studies referred to collaboration as a way to improve the relationship among learners and educators (Mas15; Pet16; Tah17). (Lau2016), as well as (Pet16), debated discussion and sharing as strategic issues for enhancing collaborative learning experiences. In addition, networking was considered by (Bas14) as a social tool, whereas other studies defined it as a learning network that generates a community of practice (Amo14; Sto15; Bar15). (Vaz17) defined networking as both a technical social tool as well as an important way to create learning communities.

For the social dimension, it was notable that the different feedback and interaction types for the support of learners by educators and by their peers were not highlighted as key social demands and instruments as identified by [28,33]. Furthermore, the studies assigned to the social dimension did not mention, and thus neglected, the specific importance of interactions with content by learners, presented as a key success factor for MOOCs by [34].

4.6. Overall Discussion across All Four Dimensions

From the overall analysis of the 103 studies across all four dimensions, it was a surprising finding that only one study addressed three or more dimensions; the other 102 studies all focused on only one or two dimensions. A potential answer could be that during the period of time analysed (publication years 2013 to 2019), studies were just beginning to analyse the quality of MOOCs and lacked general overviews and validated frameworks such as the Quality Reference Framework (QRF) [57].

Furthermore, the distribution of the 103 studies over the four dimensions was very unbalanced, as already reported in this section above, which indicates specific research interests and efforts. That was also visible in our qualitative analysis on the research focus and study content. That was not very unexpected given our main focus on pedagogical aspects and our selected search terms. Therefore, we underline the need for additional research analysis of the three under-represented dimensions.

However, we identified differences in comparison to the quantitative data. The dimension with the lowest number of studies (organisational) was addressed by a relatively high number of general studies (in particular, compared to the social dimension, with only one general study). Its flat structure of categories was not surprising, given our main focus on pedagogical aspects and in line with the technical and social dimensions.

4.7. Relevance and Future Application

We believe that this systematic literature review and its findings are relevant for MOOCs, as they inform both MOOC designers and learners. They can identify quality categories that are important for their (design as well as learning) objectives and choices of best-fitting MOOC methods. The key result of this systematic literature review, the quality framework for MOOCs, can support future MOOC research in several ways: It can be applied in MOOC design guidelines, as basis for discussions and calibration in and among MOOC design teams, adopted for standardised descriptions and quality ratings of MOOCs, and re-used in follow-up systematic literature reviews in the future.

4.8. Limitations

Some limitations must be mentioned and should be noted. First, we were not able to conduct a meta-analysis of all 103 selected studies: Too many studies would need to be excluded, as they did not provide appropriate effect sizes for a meta-analysis or gave no effect sizes at all. Second, the alternative was not even possible, that is, comparison of the provided effect sizes using the Standardized Index of Convergence (SIC) defined by [58]. The SIC can calculate the degree of consistency in findings from compared studies and can be applied when at least three studies are analysed the same relationship; it would not require comparable effect sizes [59,60,61]. However, we could not find three studies with reported effect sizes for even a single sub-category; thus, we could not apply and use the SIC method. Furthermore, we had to limit our conclusions concerning the relevance of the categories, as the numbers of studies indicated only the research interest for each category, and not necessarily the importance of each category. In addition, the limited search terms and the focus on the last seven years (2013 to 2019) led to a situation in which some relevant studies could not be selected and could have been missed in this systematic literature review. Another limitation is given through the strict focus on studies published in English and after a peer review. Finally, the quantitative as well as the qualitative analysis of the 103 studies and their assignments to the four dimensions and sub-categories revealed that there is still a great need for future in-depth research on the quality of MOOCs. A future systematic literature review could focus on the under-represented three dimensions by combining broader search terms and using only one database, to avoid search results with a high number of studies that cannot be reviewed and assigned.

5. Conclusions

In this article, the quality of MOOCs was observed and analysed from different points of view based on the results of a systematic literature review analysing 103 selected studies (see Appendix A). Our main result was the development of a quality framework as a categorisation scheme for aspects of the quality of MOOCs, distinguishing four dimensions: organisational, technical, social, and pedagogical. The pedagogical dimension was detailed and explained by a total of 33 categories on two hierarchical levels, consisting of seven first-level categories and 26 second-level categories (see Figure 1 above). The other three dimensions (see Figure A1 in Appendix B) were described by a total of only 14 first-level categories and require further research. Due to our main interest in the pedagogical aspects of MOOC quality, the other dimensions can be considered to be under-represented in this literature review compared to the pedagogical dimension, to which the vast majority of studies were assigned (see Table A1 in the Appendix C for the assignment of all 103 studies to the categories of the Quality Framework for MOOCs).

A surprising key finding from the overall analysis of the 103 studies across all four dimensions is that only one study addressed three or more dimensions. All 102 of the other studies focused on only one or two dimensions. Another finding from this systematic literature review is that it is necessary when doing research on the quality of MOOCs to consider and address several main indicators for the design and quality of MOOCs from different dimensions. Although the majority of studies focused on the pedagogical dimension, the other three dimensions (organisational, technical, and social) are also relevant and decisive for the design and quality of MOOCs. That is the case in particular for institutional commitment, which is related to the organisational dimension; social communication, belonging to the social dimension; and the provider and selected platform, which is linked to the technical dimension. These aspects were most often addressed in the studies related to each of these dimensions. These represent key research interests, but they can also be interpreted as main categories to be considered for the design and improvement of MOOCs.

MOOCs are continually becoming more and more attractive for lifelong learners, and, as a consequence, for universities and lifelong learning providers, too. MOOCs give their providers greater visibility and may allow them to address more potential students, and to enhance their purposes and impact in the context of their education, research, and involvement with other stakeholders. Therefore, it would be useful to deepen the research on the quality of MOOCs and to design new guidelines that ensure the quality of MOOCs, allowing further opening of universities to new frontiers of education.

Finally, we’ll highlight three promising directions for future research in this area. First, the results from the systematic literature review should be analysed and compared with other on-going research on the quality of MOOCs at a global level, such as the development of the Quality Reference Framework for MOOCs and open education based on the findings from the Global MOOC Quality Survey [29]. Second, the systematic literature review results should be tested and validated through other qualitative and experimental studies involving a broad range of stakeholders and institutions focusing on the design and quality of MOOCs. Third, the results should be used and evaluated in the development of new MOOCs to learn more about the practicability of the four dimensions and their quality indicators. Such mixed-method research can lead to a more comprehensive understanding and better strategies to improve the design, implementation, and evaluation of future MOOCs and therefore their quality.